Tag Archives: Health

Detecting Signs of Disease from External Images of the Eye

Posted by Boris Babenko, Software Engineer and Naama Hammel, Clinical Research Scientist, Google Health

Three years ago we wrote about our work on predicting a number of cardiovascular risk factors from fundus photos (i.e., photos of the back of the eye)1 using deep learning. That such risk factors could be extracted from fundus photos was a novel discovery and thus a surprising outcome to clinicians and laypersons alike. Since then, we and other researchers have discovered additional novel biomarkers from fundus photos, such as markers for chronic kidney disease and diabetes, and hemoglobin levels to detect anemia.

A unifying goal of work like this is to develop new disease detection or monitoring approaches that are less invasive, more accurate, cheaper and more readily available. However, one restriction to potential broad population-level applicability of efforts to extract biomarkers from fundus photos is getting the fundus photos themselves, which requires specialized imaging equipment and a trained technician.

The eye can be imaged in multiple ways. A common approach for diabetic retinal disease screening is to examine the posterior segment using fundus photographs (left), which have been shown to contain signals of kidney and heart disease, as well as anemia. Another way is to take photographs of the front of the eye (external eye photos; right), which is typically used to track conditions affecting the eyelids, conjunctiva, cornea, and lens.

In “Detection of signs of disease in external photographs of the eyes via deep learning”, in press at Nature Biomedical Engineering, we show that a deep learning model can extract potentially useful biomarkers from external eye photos (i.e., photos of the front of the eye). In particular, for diabetic patients, the model can predict the presence of diabetic retinal disease, elevated HbA1c (a biomarker of diabetic blood sugar control and outcomes), and elevated blood lipids (a biomarker of cardiovascular risk). External eye photos as an imaging modality are particularly interesting because their use may reduce the need for specialized equipment, opening the door to various avenues of improving the accessibility of health screening.

Developing the Model
To develop the model, we used de-identified data from over 145,000 patients from a teleretinal diabetic retinopathy screening program. We trained a convolutional neural network both on these images and on the corresponding ground truth for the variables we wanted the model to predict (i.e., whether the patient has diabetic retinal disease, elevated HbA1c, or elevated lipids) so that the neural network could learn from these examples. After training, the model is able to take external eye photos as input and then output predictions for whether the patient has diabetic retinal disease, or elevated sugars or lipids.

A schematic showing the model generating predictions for an external eye photo.

We measured model performance using the area under the receiver operator characteristic curve (AUC), which quantifies how frequently the model assigns higher scores to patients who are truly positive than patients who are truly negative (i.e., a perfect model scores 100%, compared to 50% for random guesses). The model detected various forms of diabetic retinal disease with AUCs of 71-82%, AUCs of 67-70% for elevated HbA1c, and AUCs of 57-68% for elevated lipids. These results indicate that, though imperfect, external eye photos can help detect and quantify various parameters of systemic health.

Much like the CDC’s pre-diabetes screening questionnaire, external eye photos may be able to help “pre-screen” people and identify those who may benefit from further confirmatory testing. If we sort all patients in our study based on their predicted risk and look at the top 5% of that list, 69% of those patients had HbA1c measurements ≥ 9 (indicating poor blood sugar control for patients with diabetes). For comparison, among patients who are at highest risk according to a risk score based on demographics and years with diabetes, only 55% had HbA1c ≥ 9, and among patients selected at random only 33% had HbA1c ≥ 9.

Assessing Potential Bias
We emphasize that this is promising, yet early, proof-of-concept research showcasing a novel discovery. That said, because we believe that it is important to evaluate potential biases in the data and model, we undertook a multi-pronged approach for bias assessment.

First, we conducted various explainability analyses aimed at discovering what parts of the image contribute most to the algorithm’s predictions (similar to our anemia work). Both saliency analyses (which examine which pixels most influenced the predictions) and ablation experiments (which examine the impact of removing various image regions) indicate that the algorithm is most influenced by the center of the image (the areas of the sclera, iris, and pupil of the eye, but not the eyelids). This is demonstrated below where one can see that the AUC declines much more quickly when image occlusion starts in the center (green lines) than when it starts in the periphery (blue lines).

Explainability analysis shows that (top) all predictions focused on different parts of the eye, and that (bottom) occluding the center of the image (corresponding to parts of the eyeball) has a much greater effect than occluding the periphery (corresponding to the surrounding structures, such as eyelids), as shown by the green line’s steeper decline. The “baseline” is a logistic regression model that takes self-reported age, sex, race and years with diabetes as input.

Second, our development dataset spanned a diverse set of locations within the U.S., encompassing over 300,000 de-identified photos taken at 301 diabetic retinopathy screening sites. Our evaluation datasets comprised over 95,000 images from 198 sites in 18 US states, including datasets of predominantly Hispanic or Latino patients, a dataset of majority Black patients, and a dataset that included patients without diabetes. We conducted extensive subgroup analyses across groups of patients with different demographic and physical characteristics (such as age, sex, race and ethnicity, presence of cataract, pupil size, and even camera type), and controlled for these variables as covariates. The algorithm was more predictive than the baseline in all subgroups after accounting for these factors.

Conclusion
This exciting work demonstrates the feasibility of extracting useful health related signals from external eye photographs, and has potential implications for the large and rapidly growing population of patients with diabetes or other chronic diseases. There is a long way to go to achieve broad applicability, for example understanding what level of image quality is needed, generalizing to patients with and without known chronic diseases, and understanding generalization to images taken with different cameras and under a wider variety of conditions, like lighting and environment. In continued partnership with academic and nonacademic experts, including EyePACS and CGMH, we look forward to further developing and testing our algorithm on larger and more comprehensive datasets, and broadening the set of biomarkers recognized (e.g., for liver disease). Ultimately we are working towards making non-invasive health and wellness tools more accessible to everyone.

Acknowledgements
This work involved the efforts of a multidisciplinary team of software engineers, researchers, clinicians and cross functional contributors. Key contributors to this project include: Boris Babenko, Akinori Mitani, Ilana Traynis, Naho Kitade‎, Preeti Singh, April Y. Maa, Jorge Cuadros, Greg S. Corrado, Lily Peng, Dale R. Webster, Avinash Varadarajan‎, Naama Hammel, and Yun Liu. The authors would also like to acknowledge Huy Doan, Quang Duong, Roy Lee, and the Google Health team for software infrastructure support and data collection. We also thank Tiffany Guo, Mike McConnell, Michael Howell, and Sam Kavusi for their feedback on the manuscript. Last but not least, gratitude goes to the graders who labeled data for the pupil segmentation model, and a special thanks to Tom Small for the ideation and design that inspired the animation used in this blog post.

1The information presented here is research and does not reflect a product that is available for sale. Future availability cannot be guaranteed. ↩

Source: Google AI Blog


The Check Up: helping people live healthier lives

My years spent caring for patients at the bedside and in the clinic inspired me to find ways to improve health for them and their communities at scale. That passion eventually brought me to Google where I could help solve the world’s most significant health challenges.

I joined the company just weeks before the COVID-19 pandemic unfolded. At the time, most people hadn’t heard of “flattening the curve” or “mRNA vaccines.” But what they did know was that they could turn to Google with their questions. The COVID-19 pandemic strengthened our resolve that Google could and should help everyone, everywhere live a healthier life. It also accelerated our company-wide health efforts.

We embed health into our products to meet people where they are. Our teams apply their expertise and technological strengths and harness the power of partnerships to support our 3Cs – consumers, caregivers and communities around the world.

Today, we’re hosting our second annual Google Health event, The Check Up. Teams from across the company — including Search, YouTube, Fitbit, Care Studio, Health AI, Cloud and Advanced Technologies and Projects team — will share updates about their latest efforts.

Among the areas of progress, I’m delighted at the ways our teams are working to support consumers with helpful information and tools throughout their health journeys.

Making it easier to find and book local care providers in the U.S.

When people have questions about their health, they often start with the internet to find answers. No matter what people are searching for on Google Search, it's our mission to give high-quality information, exactly when it’s needed.

The Search team recently released features to help people navigate the complex healthcare system and make more informed decisions, like finding healthcare providers who take their insurance.

At today's event, Hema Budaraju, who leads our Health and Social Impact work for Search, introduced a feature we’re rolling out that shows the appointment availability for healthcare providers so you can easily book an appointment. Whether you put off your annual check-up, recently moved and need a new doctor, or are looking for a same-day visit to a MinuteClinic at CVS, you might see available appointment dates and times for doctors in your area.

While we’re still in the early stages of rolling this feature out, we’re working with partners, including MinuteClinic at CVS and other scheduling solution providers. We hope to expand features, functionality and our network of partners so we can make it easier for people to get the care they need.

Screenshot of new appointment availability feature

Helping people in Brazil, India and Japan discover local, authoritative health content on YouTube

Of all the information channels people turn to for health information, video can be a helpful and powerful way to help people make informed healthcare decisions. People can watch and listen to experts translate complex medical terms and information into simple language and concepts they easily understand, and they can connect with communities experiencing similar conditions and health challenges.

Dr. Garth Graham talked about YouTube Health’s mission of providing equitable access to authoritative health information that is evidence-based, culturally relevant and engaging. In the past year, YouTube has focused on building partnerships with leading health organizations and public health leaders to increase the volume and visibility of authoritative health content through new features.

Starting this week in Japan, Brazil and India, YouTube is adding health source information panels on videos to provide context that helps viewers identify videos from authoritative sources, and health content shelves that more effectively highlight videos from these sources when people search for specific health topics. These context cues help people easily navigate and evaluate credible health information.

Supporting heart health with Fitbit

In addition to information needs, people use our consumer technologies and tools to support their health and wellness. Fitbit makes it easy and motivating for people to manage their holistic health, from activity and nutrition to sleep and mindfulness. Fitbit co-founder James Park shared how Fitbit believes wearables can have an even greater impact on supporting people with chronic conditions, including heart conditions like atrial fibrillation (AFib).

In 2020, the team launched the Fitbit Heart Study, with nearly half a million people who use Fitbit. The goal was to test our PPG (Photoplethysmography) AFib algorithm, which passively looks at heart rate data, to alert people to signs of an irregular heart rhythm.

We presented the study results at the most recent American Heart Association meeting, showing that the algorithm accurately identified undiagnosed AFib 98% of the time. We’ve submitted our algorithm to the FDA for review. This is one of many ways we’re continuing to make health even more accessible.

Building the future for better health

These updates are only a slice of what we covered at the event. Check out our Health AI blog post and tune into our event to hear more about ways we are advancing better, more equitable health for everyone.

The Check Up: our latest health AI developments

Over the years, teams across Google have focused on how technology — specifically artificial intelligence and hardware innovations — can improve access to high-quality, equitable healthcare across the globe.

Accessing the right healthcare can be challenging depending on where people live and whether local caregivers have specialized equipment or training for tasks like disease screening. To help, Google Health has expanded its research and applications to focus on improving the care clinicians provide and allow care to happen outside hospitals and doctor’s offices.

Today, at our Google Health event The Check Up, we’re sharing new areas of AI-related research and development and how we’re providing clinicians with easy-to-use tools to help them better care for patients. Here’s a look at some of those updates.

Smartphone cameras’ potential to protect cardiovascular health and preserve eyesight

One of our earliest Health AI projects, ARDA, aims to help address screenings for diabetic retinopathy — a complication of diabetes that, if undiagnosed and untreated, can cause blindness.

Today, we screen 350 patients daily, resulting in close to 100,000 patients screened to date. We recently completed a prospective study with the Thailand national screening program that further shows ARDA is accurate and capable of being deployed safely across multiple regions to support more accessible eye screenings.

In addition to diabetic eye disease, we’ve previously also shown how photos of eyes’ interiors (or fundus) can reveal cardiovascular risk factors, such as high blood sugar and cholesterol levels, with assistance from deep learning. Our recent research tackles detecting diabetes-related diseases from photos of the exterior of the eye, using existing tabletop cameras in clinics. Given the early promising results, we’re looking forward to clinical research with partners, including EyePACS and Chang Gung Memorial Hospital (CGMH), to investigate if photos from smartphone cameras can help detect diabetes and non-diabetes diseases from external eye photos as well. While this is in the early stages of research and development, our engineers and scientists envision a future where people, with the help of their doctors, can better understand and make decisions about health conditions from their own homes.

Recording and translating heart sounds with smartphones

We’ve previously shared how mobile sensors combined with machine learning can democratize health metrics and give people insights into daily health and wellness. Our feature that allows you to measure your heart rate and respiratory rate with your phone’s camera is now available on over 100 models of Android devices, as well as iOS devices. Our manuscript describing the prospective validation study has been accepted for publication.

Today, we’re sharing a new area of research that explores how a smartphone’s built-in microphones could record heart sounds when placed over the chest. Listening to someone’s heart and lungs with a stethoscope, known as auscultation, is a critical part of a physical exam. It can help clinicians detect heart valve disorders, such as aortic stenosis which is important to detect early. Screening for aortic stenosis typically requires specialized equipment, like a stethoscope or an ultrasound, and an in-person assessment.

Our latest research investigates whether a smartphone can detect heartbeats and murmurs. We're currently in the early stages of clinical study testing, but we hope that our work can empower people to use the smartphone as an additional tool for accessible health evaluation.

Partnering with Northwestern Medicine to apply AI to improve maternal health

Ultrasound is a noninvasive diagnostic imaging method that uses high-frequency sound waves to create real-time pictures or videos of internal organs or other tissues, such as blood vessels and fetuses.

Research shows that ultrasound is safe for use in prenatal care and effective in identifying issues early in pregnancy. However, more than half of all birthing parents in low-to-middle-income countries don’t receive ultrasounds, in part due to a shortage of expertise in reading ultrasounds. We believe that Google’s expertise in machine learning can help solve this and allow for healthier pregnancies and better outcomes for parents and babies.

We are working on foundational, open-access research studies that validate the use of AI to help providers conduct ultrasounds and perform assessments. We’re excited to partner with Northwestern Medicine to further develop and test these models to be more generalizable across different levels of experience and technologies. With more automated and accurate evaluations of maternal and fetal health risks, we hope to lower barriers and help people get timely care in the right settings.

To learn more about the health efforts we shared at The Check Up with Google Health, check out this blog post from our Chief Health Officer Dr. Karen DeSalvo. And stay tuned for more health-related research milestones from us.

Extending Care Studio with a new healthcare partnership

Today at the HIMSS Conference in Orlando, Florida, we’re introducing a collaboration between Google Health and MEDITECH to jointly work on an integrated clinical solution. This partnership aims to combine our data harmonization, search and summarization capabilities from Google Health’s Care Studio product suite and integrate them into their electronic health record (EHR), MEDITECH Expanse.

Health information is complex — it’s often siloed or stored across different information systems and in different formats. As a result, it can be challenging for clinicians to find the information they need all in one place and quickly make sense of it to care for their patients.

Google Health’s Care Studio technology is designed to make it easier for clinicians to find critical patient information when they need it most. Built to adhere to strict privacy controls, Care Studio works alongside electronic health records (EHRs) to enhance existing workflows. Since we launched Care Studio, we’ve continued to hone our search capabilities for medical data, notes and scanned documents, and are using AI to help make sense of clinical information. We recently introduced our Conditions feature which summarizes a patient’s conditions and uses natural language processing to link to related information — like medications or lab results — so clinicians have the context they need to understand and assess a condition.

We’re proud of what we built with Care Studio thus far, and we know that partnering is fundamental to improving health outcomes at scale — no one product or company can overcome these obstacles alone.

Collaboration with the healthcare industry

MEDITECH has made significant commitments to advancing interoperability — a commitment we share. To best support clinicians, we need to fit into the way they work now. Collaborations with EHRs, like MEDITECH, will help us seamlessly integrate Google Health tools into existing clinical workflows, so we can help remove friction for clinicians.

With MEDITECH, we’re working on a deeply integrated solution to bring some of our data harmonization, search and summarization capabilities to their web-based EHR, MEDITECH Expanse. Using Google Health’s tools, MEDITECH will form a longitudinal health data layer, bringing together data from different sources into a standard format and offering clinicians a full view of patient records. And with Google Health’s search functionality embedded into their EHR, clinicians can find salient information faster for a more frictionless experience, and the intelligent summarization can highlight critical information directly in the Expanse workflow. This will help advance healthcare data interoperability, building on MEDITECH’s vision for a more connected ecosystem. Our collaboration expands on the partnership between MEDITECH and Google Cloud and will utilize Google Cloud’s infrastructure.

The healthcare industry is at an inflection point when it comes to interoperability. As COVID accelerated the need for interoperable systems, more organizations were eager to embrace Fast Healthcare Interoperability Resources (FHIR) as the standard format for healthcare data. We’re using FHIR to support our data harmonization, yet there is more to be done before FHIR is widely adopted and systems can effectively exchange information. We’re hopeful that collaborative approaches, much like what we’re working on with MEDITECH, will help create more interoperable solutions and facilitate an open ecosystem of data interoperability that benefits everyone.

Upholding our privacy commitments

As we deepen our partnerships across the healthcare industry, privacy and security remain our top priorities. As with all our Google Health Care Studio partners, Google does not own or ever sell patient data, and patient data cannot be used for advertising. Our tools are designed to adhere to industry best practices and regulations, including HIPAA. Patient data is encrypted and isolated in a controlled access environment, separate from other Google customer data and consumer data.

Industry collaboration is a critical path to overcoming pervasive data fragmentation challenges. While we’re in the early stages with MEDITECH, this new collaboration marks an exciting step forward in creating a more open healthcare ecosystem and improving health outcomes.

Join us at HIMSS at 3:00 p.m. EST today, in room WF3 to learn more.

Take a look at Conditions, our new feature in Care Studio

At Google Health, we’re always thinking about how we can make our tools most useful for clinicians. This includes Care Studio, our clinical software that harmonizes healthcare data from different sources to give clinicians a comprehensive view of a patient’s records.

Today, at the ViVE Conference in Miami Beach, we previewed Conditions, a new Care Studio feature that helps clinicians make even better sense of patient records.

Instant insights for clinicians

Getting a holistic summary of a patient's medical history can be challenging as key clinical insights are often buried in unstructured notes and data silos. With Conditions, we use our deep understanding of data to provide a quick and concise summary of a patient’s medical conditions along with critical context from clinical notes. Conditions are organized by acuity, so a clinician can quickly determine if a patient’s condition is acute or chronic.

We also provide easy access to information related to a condition — including labs, medications, reports, specialist notes and more — to help clinicians manage and treat a condition. So if a clinician clicks on a condition, like diabetes, they may see blood sugar levels, insulin administrations, endocrinology consult notes and retinopathy screening studies. And if critical information is missing, we will highlight its absence from the chart. For example, we’d flag if standard labs for a patient with diabetes are missing, like hemoglobin A1c results. With these resources, a clinician can quickly understand a new patient’s medical history or easily review an existing patient’s insulin regimen before their appointment.

Bringing natural language processing to medical data

Healthcare data is structured in numerous ways, making it difficult to organize. Clinical notes may be written differently and stored across different systems. Clinician notes also differ based on if content is meant for clinical decision making, billing or regulatory uses. Further, when it comes to writing notes, clinicians use different abbreviations or acronyms depending on their personal preference, what health system they’re a part of, their region and other factors. All of this has made it difficult to synthesize clinical data — until now.

The Conditions feature works by algorithmically understanding medical concepts from notes that may be written in incomplete sentences, shorthand or with misspelled words. We use Google’s advances in AI in an area called natural language processing (NLP) to understand the actual context in which a condition is mentioned and map these concepts to a vocabulary of tens of thousands of medical conditions. For example: One clinician might write “multiple sclerosis exacerbation” while another might document the same problem as “MS flare”. Care Studio is able to recognize that these different terms are linked to the same condition, and supported by the same evidence.

Similarly, Care Studio understands that the statement “Patient has a history of dm”, means that diabetes mellitus (dm) is present. And for the statement “Pneumonia is not likely at this time”, pneumonia is absent.

Care Studio then ranks each condition to determine its importance using various factors — such as the condition itself, its frequency, recency and more — to elevate the most important conditions to the top. Finally, based on input from medical specialists and clinicians on the Google team, Care Studio organizes conditions to support clinical thinking and decision making. For instance, acute conditions are highlighted, and related conditions are presented next to each other.

Healthcare data is complex, and clinicians often have to manually sift through information to make sense of a patient’s conditions. We’re excited to bring this feature to clinicians in the coming months so they can instantly access the information they need all in one place to provide better care.

Machine Learning for Mechanical Ventilation Control

Posted by Daniel Suo, Software Engineer and Elad Hazan, Research Scientist, Google Research, on behalf of the Google AI Princeton Team

Mechanical ventilators provide critical support for patients who have difficulty breathing or are unable to breathe on their own. They see frequent use in scenarios ranging from routine anesthesia, to neonatal intensive care and life support during the COVID-19 pandemic. A typical ventilator consists of a compressed air source, valves to control the flow of air into and out of the lungs, and a "respiratory circuit" that connects the ventilator to the patient. In some cases, a sedated patient may be connected to the ventilator via a tube inserted through the trachea to their lungs, a process called invasive ventilation.

A mechanical ventilator takes breaths for patients who are not fully capable of doing so on their own. In invasive ventilation, a controllable, compressed air source is connected to a sedated patient via tubing called a respiratory circuit.

In both invasive and non-invasive ventilation, the ventilator follows a clinician-prescribed breathing waveform based on a respiratory measurement from the patient (e.g., airway pressure, tidal volume). In order to prevent harm, this demanding task requires both robustness to differences or changes in patients' lungs and adherence to the desired waveform. Consequently, ventilators require significant attention from highly-trained clinicians in order to ensure that their performance matches the patients’ needs and that they do not cause lung damage.

Example of a clinician-prescribed breathing waveform (orange) in units of airway pressure and the actual pressure (blue), given some controller algorithm.

In “Machine Learning for Mechanical Ventilation Control”, we present exploratory research into the design of a deep learning–based algorithm to improve medical ventilator control for invasive ventilation. Using signals from an artificial lung, we design a control algorithm that measures airway pressure and computes necessary adjustments to the airflow to better and more consistently match prescribed values. Compared to other approaches, we demonstrate improved robustness and better performance while requiring less manual intervention from clinicians, which suggests that this approach could reduce the likelihood of harm to a patient’s lungs.

Current Methods
Today, ventilators are controlled with methods belonging to the PID family (i.e., Proportional, Integral, Differential), which control a system based on the history of errors between the observed and desired states. A PID controller uses three characteristics for ventilator control: proportion (“P”) — a comparison of the measured and target pressure; integral (“I”) — the sum of previous measurements; and differential (“D”) — the difference between two previous measurements. Variants of PID have been used since the 17th century and today form the basis of many controllers in both industrial (e.g., controlling heat or fluids) and consumer (e.g., controlling espresso pressure) applications.

PID control forms a solid baseline, relying on the sharp reactivity of P control to rapidly increase lung pressure when breathing in and the stability of I control to hold the breath in before exhaling. However, operators must tune the ventilator for specific patients, often repeatedly, to balance the “ringing” of overzealous P control against the ineffectually slow rise in lung pressure of dominant I control.

Current PID methods are prone to over- and then under-shooting their target (ringing). Because patients differ in their physiology and may even change during treatment, highly-trained clinicians must constantly monitor and adjust existing methods to ensure such violent ringing as in the above example does not occur.

To more effectively balance these characteristics, we propose a neural network–based controller to create a set of control signals that are more broad and adaptable than PID-generated controls.

A Machine-Learned Ventilator Controller
While one could tune the coefficients of a PID controller (either manually or via an exhaustive grid search) through a limited number of repeated trials, it is impossible to apply such a direct approach towards a deep controller, as deep neural networks (DNNs) are often parameter-rich and require significant training data. Similarly, popular model-free approaches, such as Q-Learning or Policy Gradient, are data-intensive and therefore unsuitable for the physical system at hand. Further, these approaches don't take into account the intrinsic differentiability of the ventilator dynamical system, which is deterministic, continuous and contact-free.

We therefore adopt a model-based approach, where we first learn a DNN-based simulator of the ventilator-patient dynamical system. An advantage of learning such a simulator is that it provides a more accurate data-driven alternative to physics-based models, and can be more widely distributed for controller research.

To train a faithful simulator, we built a dataset by exploring the space of controls and the resulting pressures, while balancing against physical safety, e.g., not over-inflating a test lung and causing damage. Though PID control can exhibit ringing behavior, it performs well enough to use as a baseline for generating training data. To safely explore and to faithfully capture the behavior of the system, we use PID controllers with varied control coefficients to generate the control-pressure trajectory data for simulator training. Further, we add random deviations to the PID controllers to capture the dynamics more robustly.

We collect data for training by running mechanical ventilation tasks on a physical test lung using an open-source ventilator designed by Princeton University's People's Ventilator Project. We built a ventilator farm housing ten ventilator-lung systems on a server rack, which captures multiple airway resistance and compliance settings that span a spectrum of patient lung conditions, as required for practical applications of ventilator systems.

We use a rack-based ventilator farm (10 ventilators / artificial lungs) to collect training data for a ventilator-lung simulator. Using this simulator, we train a DNN controller that we then validate on the physical ventilator farm.

The true underlying state of the dynamical system is not available to the model directly, but rather only through observations of the airway pressure in the system. In the simulator we model the state of the system at any time as a collection of previous pressure observations and the control actions applied to the system (up to a limited lookback window). These inputs are fed into a DNN that predicts the subsequent pressure in the system. We train this simulator on the control-pressure trajectory data collected through interactions with the test lung.

The performance of the simulator is measured via the sum of deviations of the simulator’s predictions (under self-simulation) from the ground truth.

While it is infeasible to compare real dynamics with their simulated counterparts over all possible trajectories and control inputs, we measure the distance between simulation and the known safe trajectories. We introduce some random exploration around these safe trajectories for robustness.

Having learned an accurate simulator, we then use it to train a DNN-based controller completely offline. This approach allows us to rapidly apply updates during controller training. Furthermore, the differentiable nature of the simulator allows for the stable use of the direct policy gradient, where we analytically compute the gradient of the loss with respect to the DNN parameters.  We find this method to be significantly more efficient than model-free approaches.

Results
To establish a baseline, we run an exhaustive grid of PID controllers for multiple lung settings and select the best performing PID controller as measured by average absolute deviation between the desired pressure waveform and the actual pressure waveform. We compare these to our controllers and provide evidence that our DNN controllers are better performing and more robust.

  1. Breathing waveform tracking performance:

    We compare the best PID controller for a given lung setting against our controller trained on the learned simulator for the same setting. Our learned controller shows a 22% lower mean absolute error (MAE) between target and actual pressure waveforms.

    Comparison of the MAE between target and actual pressure waveforms (lower is better) for the best PID controller (orange) for a given lung setting (shown for two settings, R=5 and R=20) against our controller (blue) trained on the learned simulator for the same setting. The learned controller performs up to 22% better.
  2. Robustness:

    Further, we compare the performance of the single best PID controller across the entire set of lung settings with our controller trained on a set of learned simulators over the same settings. Our controller performs up to 32% better in MAE between target and actual pressure waveforms, suggesting that it could require less manual intervention between patients or even as a patient's condition changes.

    As above, but comparing the single best PID controller across the entire set of lung settings against our controller trained over the same settings. The learned controller performs up to 32% better, suggesting that it may require less manual intervention.

Finally, we investigated the feasibility of using model-free and other popular RL algorithms (PPO, DQN), in comparison to a direct policy gradient trained on the simulator. We find that the simulator-trained direct policy gradient achieves slightly better scores and does so with a more stable training process that uses orders of magnitude fewer training samples and a significantly smaller hyperparameter search space.

In the simulator, we find that model-free and other popular algorithms (PPO, DQN) perform approximately as well as our method.
However, these other methods take an order of magnitude more episodes to train to similar levels.

Conclusions and the Road Forward
We have described a deep-learning approach to mechanical ventilation based on simulated dynamics learned from a physical test lung. However, this is only the beginning. To make an impact on real-world ventilators there are numerous other considerations and issues to take into account. Most important amongst them are non-invasive ventilators, which are significantly more challenging due to the difficulty of discerning pressure from lungs and mask pressure. Other directions are how to handle spontaneous breathing and coughing. To learn more and become involved in this important intersection of machine learning and health, see an ICML tutorial on control theory and learning, and consider participating in one of our kaggle competitions for creating better ventilator simulators!

Acknowledgements
The primary work was based in the Google AI Princeton lab, in collaboration with Cohen lab at the Mechanical and Aerospace Engineering department at Princeton University. The research paper was authored by contributors from Google and Princeton University, including: Daniel Suo, Naman Agarwal, Wenhan Xia, Xinyi Chen, Udaya Ghai, Alexander Yu, Paula Gradu, Karan Singh, Cyril Zhang, Edgar Minasyan, Julienne LaChance, Tom Zajdel, Manuel Schottdorf, Daniel Cohen, and Elad Hazan.

Source: Google AI Blog


Group effort: How we helped launch an NYC vaccine site

In March 2021, Google initiated a considerable project: helping to distribute COVID-19 vaccines to New Yorkers in the Chelsea neighborhood of Manhattan.

Google’s New York City office is located in Chelsea, and the neighborhood is also the home to two New York City Housing Authority (NYCHA) developments, the Fulton Houses and the Elliott-Chelsea Houses. The city had identified NYCHA residents as hard-to-reach populations for vaccines — so to help, Google provided a total of more than $1 million in resources to the city to support more vaccine education and to Hudson Guild toward the creation of a local vaccination center.

“Google recognizes that equitable population vaccination is a complex problem to solve,” says Dr. Karen DeSalvo, Google’s chief health officer, “and we’re committed to doing our part.” That commitment led us to partnering with the Hudson Guild, a local nonprofit founded in 1895. Hudson Guild, which Google has worked with as a community partner for more than a decade, is a settlement house that serves 14,000 New Yorkers every year, mostly members of the Chelsea community. The nonprofit has a special relationship with local residents, and organizers and volunteers have taken their grassroots, one-on-one approach to make sure residents have the information and support they need to get vaccinated.

“The reputation you have in the community means everything. Residents who were very hesitant to get the vaccines eventually came around because we took the time to explain the science, give them reliable information and build trust,” says LeeAnn Scaduto, Hudson Guild’s chief operating officer.

A group of people standing in a line in a medical center, looking into the camera and smiling.

From left to right: Googlers Connie Choi, David Goodman, Dale Allsopp, Duncan Wood, Hudson Guild's Daisy Mendoza, Googlers Wenjie Sun and Thomas Coleman

“We’ve been able to be a consistent force in the neighborhood,” says Daisy Mendoza, director of community building for Hudson Guild. She says the team has walked the streets of the communities daily, knocked on doors, made phone calls and even stopped by local businesses to encourage owners and workers to get vaccinated. “The residents see us everyday and know we care about them,” Daisy says.

Googlers got involved in this grassroots effort, too — knocking on doors, helping in the registration efforts and serving as translators to help the vaccination site get up and running. “Many people were still isolating at that time,” says Stavroula Maliarou, a program manager at Google who helped organize the volunteer efforts. “There was a fear of being close to people who could potentially be sick. But so many Googlers showed up to help in any way they could. We know this community — and we knew they needed our help, especially at that moment.”

Of course, there were challenges. The Hudson Guild organizers said they’ve had to combat vaccine hesitancy and residents’ lack of access to technology. But they’ve overcome these obstacles thanks to relying on community volunteers and Hudson Guild staff to share information in plain language, dispel misinformation and make the vaccination process as simple as possible for recipients.

Since April 7, 2021, Hudson Guild’s Fulton Vaccine Hub, funded in part by Google, has helped vaccinate 21,250 people, 1,700 of whom are NYCHA residents. The vaccination site has been so successful it was initially extended to October 2021, and then extended indefinitely to continue bringing vaccines and boosters to the local neighborhood.

“Without Google’s help, this isn’t something we would have been able to do — this isn’t our area of expertise,” LeeAnn says. “Google gave us the opportunity to be part of the solution in a really meaningful way for our community. This allowed us to really find a solution that worked.”

If you’re interested in learning more about Hudson Guild and helping those who live, work or go to school in Chelsea and the west side of New York City, with a focus on those in need, visit hudsonguild.org.

Group effort: How we helped launch an NYC vaccine site

In March 2021, Google initiated a considerable project: helping to distribute COVID-19 vaccines to New Yorkers in the Chelsea neighborhood of Manhattan.

Google’s New York City office is located in Chelsea, and the neighborhood is also the home to two New York City Housing Authority (NYCHA) developments, the Fulton Houses and the Elliott-Chelsea Houses. The city had identified NYCHA residents as hard-to-reach populations for vaccines — so to help, Google provided a total of more than $1 million in resources to the city to support more vaccine education and to Hudson Guild toward the creation of a local vaccination center.

“Google recognizes that equitable population vaccination is a complex problem to solve,” says Dr. Karen DeSalvo, Google’s chief health officer, “and we’re committed to doing our part.” That commitment led us to partnering with the Hudson Guild, a local nonprofit founded in 1895. Hudson Guild, which Google has worked with as a community partner for more than a decade, is a settlement house that serves 14,000 New Yorkers every year, mostly members of the Chelsea community. The nonprofit has a special relationship with local residents, and organizers and volunteers have taken their grassroots, one-on-one approach to make sure residents have the information and support they need to get vaccinated.

“The reputation you have in the community means everything. Residents who were very hesitant to get the vaccines eventually came around because we took the time to explain the science, give them reliable information and build trust,” says LeeAnn Scaduto, Hudson Guild’s chief operating officer.

A group of people standing in a line in a medical center, looking into the camera and smiling.

From left to right: Googlers Connie Choi, David Goodman, Dale Allsopp, Duncan Wood, Hudson Guild's Daisy Mendoza, Googlers Wenjie Sun and Thomas Coleman

“We’ve been able to be a consistent force in the neighborhood,” says Daisy Mendoza, director of community building for Hudson Guild. She says the team has walked the streets of the communities daily, knocked on doors, made phone calls and even stopped by local businesses to encourage owners and workers to get vaccinated. “The residents see us everyday and know we care about them,” Daisy says.

Googlers got involved in this grassroots effort, too — knocking on doors, helping in the registration efforts and serving as translators to help the vaccination site get up and running. “Many people were still isolating at that time,” says Stavroula Maliarou, a program manager at Google who helped organize the volunteer efforts. “There was a fear of being close to people who could potentially be sick. But so many Googlers showed up to help in any way they could. We know this community — and we knew they needed our help, especially at that moment.”

Of course, there were challenges. The Hudson Guild organizers said they’ve had to combat vaccine hesitancy and residents’ lack of access to technology. But they’ve overcome these obstacles thanks to relying on community volunteers and Hudson Guild staff to share information in plain language, dispel misinformation and make the vaccination process as simple as possible for recipients.

Since April 7, 2021, Hudson Guild’s Fulton Vaccine Hub, funded in part by Google, has helped vaccinate 21,250 people, 1,700 of whom are NYCHA residents. The vaccination site has been so successful it was initially extended to October 2021, and then extended indefinitely to continue bringing vaccines and boosters to the local neighborhood.

“Without Google’s help, this isn’t something we would have been able to do — this isn’t our area of expertise,” LeeAnn says. “Google gave us the opportunity to be part of the solution in a really meaningful way for our community. This allowed us to really find a solution that worked.”

If you’re interested in learning more about Hudson Guild and helping those who live, work or go to school in Chelsea and the west side of New York City, with a focus on those in need, visit hudsonguild.org.

Ask a Techspert: What does AI do when it doesn’t know?

As humans, we constantly learn from the world around us. We experience inputs that shape our knowledge — including the boundaries of both what we know and what we don’t know.

Many of today’s machines also learn by example. However, these machines are typically trained on datasets and information that doesn’t always include rare or out-of-the-ordinary examples that inevitably come up in real-life scenarios. What is an algorithm to do when faced with the unknown?

I recently spoke with Abhijit Guha Roy, an engineer on the Health AI team, and Ian Kivlichan, an engineer on the Jigsaw team, to hear more about using AI in real-world scenarios and better understand the importance of training it to know when it doesn’t know.

Abhijit, tell me about your recent research in the dermatology space.

We’re applying deep learning to a number of areas in health, including in medical imaging where it can be used to aid in the identification of health conditions and diseases that might require treatment. In the dermatological field, we have shown that AI can be used to help identify possible skin issues and are in the process of advancing research and products, including DermAssist, that can support both clinicians and people like you and me.

In these real-world settings, the algorithm might come up against something it's never seen before. Rare conditions, while individually infrequent, might not be so rare in aggregate. These so-called “out-of-distribution” examples are a common problem for AI systems which can perform less well when it’s exposed to things they haven’t seen before in its training.

Can you explain what “out-distribution” means for AI?

Most traditional machine learning examples that are used to train AI deal with fairly unsubtle — or obvious — changes. For example, if an algorithm that is trained to identify cats and dogs comes across a car, then it can typically detect that the car — which is an “out-of-distribution” example — is an outlier. Building an AI system that can recognize the presence of something it hasn’t seen before in training is called “out-of-distribution detection,” and is an active and promising field of AI research.

Okay, let’s go back to how this applies to AI in medical settings.

Going back to our research in the dermatology space, the differences between skin conditions can be much more subtle than recognizing a car from a dog or a cat, even more subtle than recognizing a previously unseen “pick-up truck” from a “truck”. As such, the out-of-distribution detection task in medical AI demands even more of our focused attention.

This is where our latest research comes in. We trained our algorithm to recognize even the most subtle of outliers (a so-called “near-out of distribution” detection task). Then, instead of the model inaccurately guessing, it can take a safer course of action — like deferring to human experts.

Ian, you’re working on another area where AI needs to know when it doesn’t know something. What’s that?

The field of content moderation. Our team at Jigsaw used AI to build a free tool called Perspective that scores comments according to how likely they are to be considered toxic by readers. Our AI algorithms help identify toxic language and online harassment at scale so that human content moderators can make better decisions for their online communities. A range of online platforms use Perspective more than 600 million times a day to reduce toxicity and the human time required to moderate content.

In the real world, online conversations — both the things people say and even the ways they say them — are continually changing. For example, two years ago, nobody would have understood the phrase “non-fungible token (NFT).” Our language is always evolving, which means a tool like Perspective doesn't just need to identify potentially toxic or harassing comments, it also needs to “know when it doesn’t know,” and then defer to human moderators when it encounters comments very different from anything it has encountered before.

In our recent research, we trained Perspective to identify comments it was uncertain about and flag them for separate human review. By prioritizing these comments, human moderators can correct more than 80% of the mistakes the AI might otherwise have made.

What connects these two examples?

We have more in common with the dermatology problem than you'd expect at first glance — even though the problems we try to solve are so different.

Building AI that knows when it doesn’t know something means you can prevent certain errors that might have unintended consequences. In both cases, the safest course of action for the algorithm entails deferring to human experts rather than trying to make a decision that could lead to potentially negative effects downstream.

There are some fields where this isn’t as important and others where it’s critical. You might not care if an automated vegetable sorter incorrectly sorts a purple carrot after being trained on orange carrots, but you would definitely care if an algorithm didn’t know what to do about an abnormal shadow on an X-ray that a doctor might recognize as an unexpected cancer.

How is AI uncertainty related to AI safety?

Most of us are familiar with safety protocols in the workplace. In safety-critical industries like aviation or medicine, protocols like “safety checklists” are routine and very important in order to prevent harm to both the workers and the people they serve.

It’s important that we also think about safety protocols when it comes to machines and algorithms, especially when they are integrated into our daily workflow and aid in decision-making or triaging that can have a downstream impact.

Teaching algorithms to refrain from guessing in unfamiliar scenarios and to ask for help from human experts falls within these protocols, and is one of the ways we can reduce harm and build trust in our systems. This is something Google is committed to, as outlined in its AI Principles.

Advancing genomics to better understand and treat disease

Genome sequencing can help us better understand, diagnose and treat disease. For example, healthcare providers are increasingly using genome sequencing to diagnose rare genetic diseases, such as elevated risk for breast cancer or pulmonary arterial hypertension, which are estimated to affect roughly 8% of the population.

At Google Health, we’re applying our technology and expertise to the field of genomics. Here are recent research and industry developments we’ve made to help quickly identify genetic disease and foster the equity of genomic tests across ancestries. This includes an exciting new partnership with Pacific Biosciences to further advance genomic technologies in research and the clinic.

Helping identify life-threatening disease when minutes matter

Genetic diseases can cause critical illness, and in many cases, a timely identification of the underlying issue can allow for life-saving intervention. This is especially true in the case of newborns. Genetic or congenital conditions affect nearly 6% of births, but clinical sequencing tests to identify these conditions typically take days or weeks to complete.

We recently worked with the University of California Santa Cruz Genomics Institute to build a method – called PEPPER-Margin-DeepVariant – that can analyze data for Oxford Nanopore sequencers, one of the fastest commercial sequencing technologies used today. This week, the New England Journal of Medicine published a study led by the Stanford University School of Medicine detailing the use of this method to identify suspected disease-causing variants in five critical newborn intensive care unit (NICU) cases.

In the fastest cases, a likely disease-causing variant was identified less than 8 hours after sequencing began, compared to the prior fastest time of 13.5 hours. In five cases, the method influenced patient care. For example, the team quickly turned around a diagnosis of Poirier–Bienvenu neurodevelopmental disorder for one infant, allowing for timely, disease-specific treatment.

Time required to sequence and analyze individuals in the pilot study. Disease-causing variants were identified in patient IDs 1, 2, 8, 9, and 11.

Applying machine learning to maximize the potential in sequencing data

Looking forward, new sequencing instruments can lead to dramatic breakthroughs in the field. We believe machine learning (ML) can further unlock the potential of these instruments. Our new research partnership with Pacific Biosciences (PacBio), a developer of genomic sequence platforms, is a great example of how Google’s machine learning and algorithm development tools can help researchers unlock more information from sequencing data.

PacBio’s long-read HiFi sequencing provides the most comprehensive view of genomes, transcriptomes and epigenomes. Using PacBio’s technology in combination with DeepVariant, our award-winning variant detection method, researchers have been able to accurately identify diseases that are otherwise difficult to diagnose with alternative methods.

Additionally, we developed a new open source method called DeepConsensus that, in combination with PacBio’s sequencing platforms, creates more accurate reads of sequencing data. This boost in accuracy will help researchers apply PacBio’s technology to more challenges, such as the final completion of the Human Genome and assembling the genomes of all vertebrate species.

Supporting more equitable genomics resources and methods

Like other areas of health and medicine, the genomics field grapples with health equity issues that, if not addressed, could exclude certain populations. For example, the overwhelming majority of participants in genomic studies have historically been of European ancestry. As a result, the genomics resources that scientists and clinicians use to identify and filter genetic variants and to interpret the significance of these variants are not equally powerful across individuals of all ancestries.

In the past year, we’ve supported two initiatives aimed at improving methods and genomics resources for under-represented populations. We collaborated with 23andMe to develop an improved resource for individuals of African ancestry, and we worked with the UCSC Genomics Institute to develop pangenome methods with this work recently published in Science.

In addition, we recently published two open-source methods that improve genetic discovery by more accurately identifying disease labels and improving the use of health measurements in genetic association studies.

We hope that our work developing and sharing these methods with those in the field of genomics will improve overall health and the understanding of biology for everyone. Working together with our collaborators, we can apply this work to real-world applications.