Tag Archives: accessibility

How we build with and for people with disabilities

Editor’s note: Today is Global Accessibility Awareness Day. We’re also sharing how we’re making education more accessibleand launching a newAndroid accessibility feature.

Over the past nine years, my job has focused on building accessible products and supporting Googlers with disabilities. Along the way, I’ve been constantly reminded of how vast and diverse the disability community is, and how important it is to continue working alongside this community to build technology and solutions that are truly helpful.

Before delving into some of the accessibility features our teams have been building, I want to share how we’re working to be more inclusive of people with disabilities to create more accessible tools overall.

Nothing about us, without us

In the disability community, people often say “nothing about us without us.” It’s a sentiment that I find sums up what disability inclusion means. The types of barriers that people with disabilities face in society vary depending on who they are, where they live and what resources they have access to. No one’s experience is universal. That’s why it’s essential to include a wide array of people with disabilities at every stage of the development process for any of our accessibility products, initiatives or programs.

We need to work to make sure our teams at Google are reflective of the people we’re building for. To do so, last year we launched our hiring site geared toward people with disabilities — including our Autism Career Program to further grow and strengthen our autistic community. Most recently, we helped launch the Neurodiversity Career Connector along with other companies to create a job portal that connects neurodiverse candidates to companies that are committed to hiring more inclusively.

Beyond our internal communities, we also must partner with communities outside of Google so we can learn what is truly useful to different groups and parlay that understanding into the improvement of current products or the creation of new ones. Those partnerships have resulted in the creation of Project Relate, a communication tool for people with speech impairments, the development of a completely new TalkBack, Android’s built-in screen reader, and the improvement of Select-to-Speak, a Chromebook tool that lets you hear selected text on your screen spoken out loud.

Equitable experiences for everyone

Engaging and listening to these communities — inside and outside of Google — make it possible to create tools and features like the ones we’re sharing today.

The ability to add alt-text, which is a short description of an image that is read aloud by screen readers, directly to images sent through Gmail starts rolling out today. With this update, people who use screen readers will know what’s being sent to them, whether it’s a GIF celebrating the end of the week or a screenshot of an important graph.

Communication tools that are inclusive of everyone are especially important as teams have shifted to fully virtual or hybrid meetings. Again, everyone experiences these changes differently. We’ve heard from some people who are deaf or hard of hearing, that this shift has made it easier to identify who is speaking — something that is often more difficult in person. But, in the case of people who use ASL, we’ve heard that it can be difficult to be in a virtual meeting and simultaneously see their interpreter and the person speaking to them.

Multi-pin, a new feature in Google Meet, helps solve this. Now you can pin multiple video tiles at once, for example, the presenter’s screen and the interpreter’s screen. And like many accessibility features, the usefulness extends beyond people with disabilities. The next time someone is watching a panel and wants to pin multiple people to the screen, this feature makes that possible.

We've also been working to make video content more accessible to those who are blind or low-vision through audio descriptions that describe verbally what is on the screen visually. All of our English language YouTube Originals content from the past year — and moving forward — will now have English audio descriptions available globally. To turn on the audio description track, at the bottom right of the video player, click on “Settings”, select “Audio track”, and choose “English descriptive”.

For many people with speech impairments, being understood by the technology that powers tools like voice typing or virtual assistants can be difficult. In 2019, we started work to change that through Project Euphonia, a research initiative that works with community organizations and people with speech impairments to create more inclusive speech recognition models. Today, we’re expanding Project Euphonia’s research to include four more languages: French, Hindi, Japanese and Spanish. With this expansion, we can create even more helpful technology for more people — no matter where they are or what language they speak.

I’ve learned so much in my time working in this space and among the things I’ve learned is the absolute importance of building right alongside the very people who will most use these tools in the end. We’ll continue to do that as we work to create a more inclusive and accessible world, both physically and digitally.

Helping every student learn how they learn best

Editor’s note: Today is Global Accessibility Awareness Day. We’re also sharing how we’re partnering with people with disabilitiesto build products and a newAndroid accessibility feature.

I often think about what Laura Allen, a Googler who leads our accessibility and disability inclusion work and is low vision, shared with me about her experience growing up using assistive technology in school. She said: “Technology should help children learn the way they need to learn, it shouldn’t be a thing that makes them feel different in the classroom.”

As someone who has spent years building technology at Google, I’ve thought a lot about how we can create the best possible experience for everyone. A big part of getting that right is building accessibility right into our products — which is especially important when it comes to technology that helps students learn. Ninety-five percent of students who have disabilities attend traditional schools, but the majority of those classrooms lack resources to support their needs. The need for accessible learning experiences only intensifies with the recent rise of blended learning environments.

We want students to have the tools they need to express themselves and access information in a way that works best for them. Here are a few recent ways we’ve built accessibility features directly into our education tools.

  • You can now add alt-text in Gmail. This allows people to add context for an image, making it accessible for people using screen readers and helping them better understand exactly what is being shared.
  • We’ve improved our Google Docs experience with braille support. With comments and highlights in braille, students reading a Google Doc will now hear start and end indications for comments and highlights alongside the rest of the text. This change makes it easier for people using screen readers and refreshable braille displays to interact with comments in documents and identify text with background colors.

We added new features to dictation on Chrome OS. Now you canspeak into any text field on the Chromebook simply by clicking on the mic icon in the status area or pressing Search + d to dictate. The dictation feature can be helpful for students who have trouble writing — whether that's because of dysgraphia, having a motor disability or something else. You can also edit using just your voice. Simply say “new line” to move the cursor to another line, “help” to see the full list of commands, or “undo” to fix any typos or mistakes.

Accessibility in action

We see the helpfulness of these features when they’re in the hands of teachers and students. My team recently spoke with Tracey Green, a teacher of the Deaf and an Itinerant Educational Specialist from the Montreal Oral School for the Deaf (MOSD) in Quebec. Her job is to work with students with hearing loss who attend local schools.

She and Chris Webb, who is a teacher at John Rennie High School and also a Google for Education Certified Innovator and Trainer, have been using Google Classroom to support students throughout distance learning and those who have returned to the classroom. For example, they integrate YouTube videos with automatic captioning and rely on captions in Google Meet. Their efforts to improve access to information during school assemblies kicked off a school-wide, student-led accessibility initiative to raise awareness about hearing loss and related accessibility issues.

Benefiting everyone

One phenomenon that underscores how disability-first features benefit everyone is called the “Curb-cut Effect.” When curbs were flattened to allow access for people with disabilities, it also meant greater access for bikers, skateboarders, and people pushing strollers or shopping carts. Everyone benefitted. Similarly, accessibility improvements like these recent updates to our education tools mean a better experience for everyone.

We see this similar effect time and time again among our own products. Take Live Caption in the Chrome browser for example. Similar to Google Meet captions, Live Caption in Chrome captions any video and audio content on your browser, which can be especially helpful for students who are deaf or hard of hearing. It can also be helpful when people want to read content without noise so they don’t disrupt the people around them.

When we build accessible products, we build for everyone. It’s one of the things I love about working for Google — that we serve the world. There’s a lot of work ahead of us to make sure our products delight all people, with and without disabilities. I’m excited and humbled by technology’s potential to help get us closer to this future.

Stay up-to-date on the latest accessibility features from Google for Education.

Making Android more accessible for braille users

Editor’s note: Today is Global Accessibility Awareness Day, and we’ll be sharing more on how we’re partnering with people with disabilitiesand what we’re doing to make education more accessible.

The heart of our mission at Google is making the world’s information truly accessible. But the reality is we can only realize this mission with the help of the community. This year at I/O, we announced one more step in the right direction, thanks to feedback and help from our users: We’re making it easier for braille readers to use Android. Available in our next Android 13 Beta in a few weeks, we are beginning to build out-of-the-box support for braille displays in Talkback, our screen reader within Android.

A refreshable braille display is an electro-mechanical device that creates braille patterns by raising rounded pins through holes in a flat surface. Braille-literate computer users use the braille display to touch-read braille dots representing text. With the display, you can also type out braille. These devices help people with deafblindness access mobile phones and people with blindness use their phones silently. Previously, people connected their Android devices to braille displays using the BrailleBack app, which required a separate download from the Play Store, or used a virtual keyboard within Talkback instead of a physical device.

With this new update, there are no additional downloads necessary to use most braille displays. People can use braille displays to access many of the same features available with Talkback. For instance, you can use display buttons to navigate your screen and then do activities like compose an email, make a phone call, send a text message or read a book.

There are also new shortcuts that make it easier to use braille displays with Talkback. Now there are shortcuts for navigating so it’s easier to scroll and move to the next character, word or line. There are also shortcuts for settings and for editing, like jumping to the end of documents or selecting, copying and pasting.

You can sign up for the Android beta program to try out Talkback 13 in the next beta release.

We are grateful to the community for their ongoing feedback that makes features like these possible. This is just the first step forward in developing this integration, and we can’t wait to do even more to expand the feature and to create even more related capabilities.

Improved announcements for braille comments and highlights available in Google Docs on Web

Quick summary 

We've improved how comments and highlighted text are announced with braille support enabled in Google Docs. This change makes it easier for users of assistive technology, including screen readers and refreshable braille displays, to interact with comments in documents and identify text with background colors. 

When reading the document, you’ll now hear start and end indications for comments and highlights alongside the rest of the text. These announcements will respect the comment and marked text settings that screen readers provide. We hope this additional information serves as useful context and makes it easier to collaborate when working in Google Docs. 

Getting started 

  • Admins: There is no admin control for this feature. 
  • End users: This feature will be OFF by default and can be enabled by the user going to Tools > Accessibility settings > Turn on braille support. Visit the Help Center to learn more about how to Use a braille display with Docs editors. 

Additional details 

Users should update to the latest versions of their assistive technologies and browsers to fully benefit from these improvements. 

Rollout pace 


Available to all Google Workspace customers, as well as legacy G Suite Basic and Business customers


What World Hearing Day means for this Googler

Dimitri Kanevsky, a research scientist at Google with an extensive background in mathematics, knows the impact technology can have when built with accessibility in mind. Having lost his hearing in early childhood, he imagines a world where technology can make it easier for people who are deaf or hard of hearing to be a part of everyday, in-person conversations with hearing people. Whether it's ordering coffee at a cafe, conversing with coworkers or checking out at the grocery store.

Dimitri has been turning that idea into a reality. He co-created Live Transcribe, our speech-to-text technology, which launched in 2019 and is now used daily by over a million people to communicate — including Dimitri. He works closely with the team to develop new and helpful features — like an offline mode that will be launching in the coming weeks to give people access to real-time captions even when Wi-Fi and data are unavailable.

For World Hearing Day, we talked with Dimitri about his work, why building for everyone matters and the future of accessible technology.

Tell us more about your background and job at Google.

When I moved to the U.S in 1984, there were no transcription services. I wanted to change that, so I focused my work on optimizing speech and language recognition to help people who are deaf or hard of hearing.

I eventually moved from academia to Google’s speech recognition team in 2014. The work my team and I accomplished allowed us to create practical applications — like Live Transcribe and Live Caption.

How has your personal experience shaped your career?

I completely lost my hearing when I was one. I learned to lipread well so I could communicate with other students and teachers. My family was also very helpful to me. When I switched to a school where my father taught, he made sure I was in a class with children I knew so it was a smoother transition.

But in eighth grade, I moved to a math school with new teachers and students and was unable to lipread what they taught in class or communicate with my new classmates. I sat, day after day, not understanding the material they were teaching and had to teach myself from textbooks. If I had a tool like Live Transcribe when I was growing up, my experience would have been very different.

In what ways has assistive technology — like Live Transcribe — changed your experience today?

Technology provides tremendous opportunities to help people with disabilities — I know this firsthand.

I use Live Transcribe every day to communicate with others. I use it to play games and share stories with my twin granddaughters — which is life-changing. And just last week, I gave a lecture at a mathematical seminar at John Hopkins University. During it, I could interact with the audience and answer questions — without Live Transcribe that would have been very difficult for me to do.

I used to rely heavily on lipreading for day-to-day tasks, but when people wear masks I can't do that — I don't even know when someone who's wearing a mask is talking to me. Because of this, Live Transcribe is even more important to me — especially when at stores, riding public transit or visiting a doctor.

What are you excited about when you think about speech recognition technology ten years from now?

My dream is to use speech recognition technology to help people communicate. As technology advances, it will unlock new possibilities — such as transcribing speech even as people switch languages, understanding people with all accents and speech motor skills, indicating more sound events with visual symbols and automatically integrating sign recognition or additional haptic feedback technologies.

Further in the future, I hope to see an experience where people are no longer dependent on a mobile phone to see transcriptions. Perhaps transcriptions will be available in convenient wearable eye technologies or appear on a wall when someone looks at it. There's a variant of prediction that there will be no mobile phones since all devices around us — like our walls — will act as mobile devices when people need them to.

What do you want others to learn from World Hearing Day?

According to WHO, one in ten people will experience hearing loss by 2050. Still, a lot of people with hearing loss don’t know about novel speech recognition technologies that could help them communicate, and hearing people aren’t aware of these tools.

World Hearing Day is an opportunity to make everybody aware of the needs of people with hearing loss and the technology that everyone can use to have a tremendous impact on their lives.

Google turns purple for International Day of Persons with Disabilities

Over one billion people worldwide have some form of disability: that’s one in seven. Many of those disabilities are invisible, while others can affect any of us at any time in our lives.

Today, our offices in Zurich, London, Wroclaw and Munich, and the Hyperlink Bridge in Dublin that connects three Google buildings, will light up purple to celebrate International Day of People with Disabilities (IDPwD). We will also be arranging to light up a city monument in Nairobi, Kenya. This United Nations observed day is aimed at increasing public awareness, understanding, and acceptance of people with any form of disability.

With this initiative we join #PurpleLightUp, a global movement started by PurpleSpace.org that celebrates and draws attention to the economic contribution of the 386 million disabled employees around the world. Since 2017, #PurpleLightUp has been driving momentum for disability inclusion across many organisations, with initiatives that span from hosting employee events and workshops, to lighting up iconic buildings purple, from developing new workplace policies to sparking conversations about disability inclusion.

Google became a member of PurpleSpace earlier this year, as a result of the commitment of the many Googlers from our employee resource group the Disability Alliance in Europe, the Middle East and Africa. Working with PurpleSpace will help us further raise awareness of the unique value of the disabled community, encouraging people to be more and more inclusive.

There’s a great opportunity for us to change perceptions, to destigmatize what it means to have a disability, to allow people to see all the diverse perspectives of who we are, and to amplify the value people with disabilities bring. It takes devotion and intention, but together, we can really make a difference.

You’re invited to join us in celebrating #PurpleLightUp day this year. Find out how your organisation can get involved by visiting PurpleSpace.org.

Machine learning to make sign language more accessible

Google has spent over twenty years helping to make information accessible and useful in more than 150 languages. And our work is definitely not done, because the internet changes so quickly. About 15% of searches we see are entirely new every day. And when it comes to other types of information beyond words, in many ways, technology hasn’t even begun to scratch the surface of what’s possible. Take one example: sign language.

The task is daunting. There are as many sign languages as there are spoken languages around the world. That’s why, when we started exploring how we could better support sign language, we started small by researching and experimenting with what machine learning models could recognize. We also spoke with members of the Deaf community, as well as linguistic experts. We began combining several ML models to recognize sign language as a sum of its parts — going beyond just hands to include body gestures and facial expressions.

After 14 months of testing with a database of videos for Japanese Sign Language and Hong Kong Sign Language, we launched SignTown: an interactive desktop application that works with a web browser and camera.

SignTown is an interactive web game built to help people to learn about sign language and Deaf culture. It uses machine learning to detect the user's ability to perform signs learned from the game.

Project Shuwa

SignTown is only one component of a broader effort to push the boundaries of technology for sign language and Deaf culture, named “Project Shuwa” after the Japanese word for sign language (“手話”). Future areas of development we’re exploring include building a more comprehensive dictionary across more sign and written languages, as well as collaborating with the Google Search team on surfacing these results to improve search quality for sign languages.

A woman in a black top facing the camera and making a sign with her right hand.

Advances in AI and ML now allow us to reliably detect hands, body poses and facial expressions using any camera inside a laptop or mobile phone. SignTown uses the MediaPipe Holistic model to identify keypoints from raw video frames, which we then feed into a classifier model to determine which sign is the closest match. This all runs inside of the user's browser, powered by Tensorflow.js.

A grid with separate images of four people facing the camera and making signs with their hands.

We open-sourced the core models and tools for developers and researchers to build their own custom models at Google IO 2021. That means anyone who wants to train and deploy their own sign language model has the ability to do so.

At Google, we strive to help build a more accessible world for people with disabilities through technology. Our progress depends on collaborating with the right partners and developers to shape experiments that may one day become stand-alone tools. But it’s equally important that we raise awareness in the wider community to foster diversity and inclusivity. We hope our work in this area with SignTown gets us a little closer to that goal.

More accessible web images arrive in 10 new languages

Images can be an integral part of many people’s online experiences. We rely on them to help bring news stories to life, see what our family and friends are up to, or help us decide which couch to buy. However, for 338 million people who are blind or have moderate to severe vision impairment, knowing what's in a web image that isn’t properly labeled can be a challenge. Screen reader technology relies on the efforts of content creators and developers who manually label images in order to make them accessible through spoken feedback or braille. Yet, billions of web images remain unlabelled, rendering them inaccessible for these users.

To help close this gap, the Chrome Accessibility and Google Research teams collaborated on developing a feature that automatically describes unlabelled images using AI. This feature was first released in 2019 supporting English only and was subsequently extended to five new languages in 2020 – French, German, Hindi, Italian and Spanish.

Today, we are expanding this feature to support ten additional languages: Croatian, Czech, Dutch, Finnish, Indonesian, Norwegian, Portuguese, Russian, Swedish and Turkish.

The major innovation behind this launch is the development of a single machine learning model that generates descriptions in each of the supported languages. This enables a more equitable user experience across languages in the sense that the generated image descriptions in any two languages can often be regarded as translations that respect the image details (Thapliyal and Soricut (2020)).

group of friends jumping on the beach

Auto-generated image descriptions can be incredibly helpful and their quality has come a long way, but it’s important to note they still can’t caption all images as well as a human. Our system was built to describe natural images and is unlikely to generate a description for other types of images, such as sketches, cartoons, memes or screenshots. We considered fairness, safety and quality when developing this feature and implemented a process to evaluate the images and captions along these dimensions before they're eligible to be shown to users.

We are excited to take this next step towards improving accessibility for more people around the world and look forward to expanding support to more languages in the future.

To activate this feature, you first need to turn on your screen reader (here's how to do that in Chrome). From there, you can activate the “Get image descriptions from Google” feature either by opening the context menu when browsing a web page or under your browser’s Accessibility settings. Chrome will then automatically generate descriptions for unlabelled web images in your preferred language.

Predicting Text Readability from Scrolling Interactions

Illiteracy affects at least 773 million people globally, both young and old. For these individuals, reading information from unfamiliar sources or on unfamiliar topics can be extremely difficult. Unfortunately, these inequalities have been further magnified by the global pandemic as a result of unequal access to education in reading and writing. In fact, UNESCO reports that over 100 million children are falling behind the minimum proficiency level in reading due to COVID-related school closures.

With increasing world-wide access to technology, reading on a device, such as a tablet or phone, has largely taken the place of traditional formats. This provides a unique opportunity to observe reading interactions, e.g., how a reader scrolls through a text, which can inform our understanding of what can make text difficult to read. This understanding is crucial when designing educational applications for low-proficiency readers and language learners, because it can be used to match learners with appropriately leveled texts as well as to support readers in understanding texts beyond their reading level.

In “Predicting Text Readability from Scrolling Interactions”, presented at CoNLL 2021, we show that data from on-device reading interactions can be used to predict how readable a text is. This novel approach provides insights into subjective readability — whether an individual reader has found a text accessible — and demonstrates that existing readability models can be improved by including feedback from scroll-based reading interactions. In order to encourage research in this area and to help enable more personalized tools for language learning and text simplification, we are releasing the dataset of reading interactions generated from our scrolling behavior–based readability assessment of English-language texts.

Understanding Text Difficulty
There are multiple aspects of a text that impact how difficult it is to read, including the vocabulary level, the syntactic structure, and overall coherence. Traditional machine learning approaches to measure readability have exclusively relied on such linguistic features. However, using these features alone does not work well for online content, because such content often contains abbreviations, emojis, broken text, and short passages, which detrimentally impact the performance of readability models.

To address this, we investigated whether aggregate data about the reading interactions of a group can be used to predict how difficult a text is, as well as how reading interactions may differ based on a readers’ understanding. When reading on a device, readers typically interact with text by scrolling in a vertical fashion, which we hypothesize can be used as a coarse proxy for reading comprehension. With this in mind, we recruited 518 paid participants and asked them to read English-language texts of different difficulty levels. We recorded the reading interactions by measuring different features of the participants’ scrolling behavior, such as the speed, acceleration and number of times areas of text were revisited. We then used this information to produce a set of features for a readability classifier.

Predicting Text Difficulty from Scrolling Behavior
We investigated which types of scrolling behaviors were most impacted by text difficulty and tested the significance using linear mixed effect models. In our set up, we have repeated measures, as multiple participants read the same texts and each participant reads more than one text. Using linear mixed-effect models gives us a higher confidence that the differences in interactions we are observing are because of the text difficulty, and not other random effects.

Our results showed that multiple reading behaviors differed significantly based on the text level, for example, the average, maximum and minimum acceleration of scrolling. We found the most significant features to be the total read time and the maximum reading speeds.

We then used these features as inputs to a machine learning algorithm. We designed and trained a support vector machine (i.e., a binary classifier) to predict whether a text is either advanced or elementary based only on scrolling behaviors as individuals interacted with it. The dataset on which the model was trained contains 60 articles, each of which were read by an average of 17 participants. From these interactions we produced aggregate features by taking the mean of the significant measures across participants.


We measured the accuracy of the approach using a metric called f-score, which measures how accurate the model is at classifying a text as either “easy” or “difficult” (where 1.0 reflects perfect classification accuracy). We are able to achieve an f-score of 0.77 on this task, using interaction features alone. This is the first work to show that it is possible to predict the readability of a text using only interaction features.

Improving Readability Models
In order to demonstrate the value of applying readability measures from scrolling behaviors to existing readability models, we integrated scroll-based features into the state-of-the-art automated readability assessment tool, which was released as part of the OneStopEnglish corpus. We found that the addition of interaction features improves the f-score of this model from 0.84 to 0.88. In addition, we were able to significantly outperform this system by using interaction information with simple vocabulary features, such as the number of words in the text, achieving an impressive f-score of 0.96.

In our study, we recorded comprehension scores to evaluate the understanding and readability of text for individuals. Participants were asked three questions per article to assess the reader’s understanding of what they had read. The interaction features of an individual’s scrolling behavior was represented as a high dimensional vector. To explore this data, we visualized the reading interaction features for each participant using t-distributed stochastic neighbor embeddings, which is a statistical method for visualizing high-dimensional data. The results revealed clusters in the comprehension score based on how well individuals understood the text. This shows that there is implicit information in reading interactions about the likelihood that an individual has understood a given text. We refer to this phenomenon as subjective readability. This information can be very useful for educational applications or for simplifying online content.

Plot showing t-SNE projection of scroll interactions in 2-dimensions. The color of each data point corresponds to the comprehension score. Clusters of comprehension scores indicate that there are correlations between reading behaviors and comprehension.

Finally, we investigated the extent to which reading interactions vary across audiences. We compared the average scrolling speed across different reader groups, covering reading proficiency and the reader’s first language. We found that the speed distribution varies depending on the proficiency and first language of the audience. This supports the case that first language and proficiency alter the reading behaviors of audiences, which allows us to contextualize the reading behavior of groups and better understand which areas of text may be harder for them to read.

Histogram showing the average speeds of scrolling (in vertical pixels per millisecond) across readers of different proficiency levels (beginner, intermediate and advanced), with lines showing the smoothed trend for each group. A higher average scroll speed indicates faster reading times. For example, a more challenging text that corresponds to slower scroll speeds by advanced readers is associated with higher scroll speeds by beginners because they engage with the text only superficially.

Histogram showing the average speeds of scrolling (in vertical pixels per millisecond) across audiences by first language of the readers, Tamil or English, with lines showing the smoothed trend for each group. A higher average scroll speed indicates faster reading times. Dark blue bars are where the histograms overlap.

This work is the first to show that reading interactions, such as scrolling behavior, can be used to predict the readability of text, which can yield numerous benefits. Such measures are language agnostic, unobtrusive, and robust to noisy text. Implicit user feedback allows insight into readability at an individual level, thereby allowing for a more inclusive and personalisable assessment of text difficulty. Furthermore, being able to judge the subjective readability of text benefits language learning and educational apps. We conducted a 518 participant study to investigate the impact of text readability on reading interactions and are releasing a novel dataset of the associated reading interactions. We confirm that there are statistically significant differences in the way that readers interact with advanced and elementary texts, and that the comprehension scores of individuals correlate with specific measures of scrolling interaction. For more information our conference presentation is available to view.

We thank our collaborators Yevgeni Berzak, Tony Mak and Matt Sharifi, as well as Dmitry Lagun and Blaise Aguera y Arcas for their helpful feedback on the paper.

Source: Google AI Blog

An Open Source Vibrotactile Haptics Platform for On-Body Applications.

Most wearable smart devices and mobile phones have the means to communicate with the user through tactile feedback, enabling applications from simple notifications to sensory substitution for accessibility. Typically, they accomplish this using vibrotactile actuators, which are small electric vibration motors. However, designing a haptic system that is well-targeted and effective for a given task requires experimentation with the number of actuators and their locations in the device, yet most practical applications require standalone on-body devices and integration into small form factors. This combination of factors can be difficult to address outside of a laboratory as integrating these systems can be quite time-consuming and often requires a high level of expertise.

A typical lab setup on the left and the VHP board on the right.

In “VHP: Vibrotactile Haptics Platform for On-body Applications”, presented at ACM UIST 2021, we develop a low-power miniature electronics board that can drive up to 12 independent channels of haptic signals with arbitrary waveforms. The VHP electronics board can be battery-powered, and integrated into wearable devices and small gadgets. It allows all-day wear, has low latency, battery life between 3 and 25 hours, and can run 12 actuators simultaneously. We show that VHP can be used in bracelet, sleeve, and phone-case form factors. The bracelet was programmed with an audio-to-tactile interface to aid lipreading and remained functional when worn for multiple months by developers. To facilitate greater progress in the field of wearable multi-channel haptics with the necessary tools for their design, implementation, and experimentation, we are releasing the hardware design and software for the VHP system via GitHub.

Front and back sides of the VHP circuit board.
Block diagram of the system.

Platform Specifications.
VHP consists of a custom designed circuit board, where the main components are the microcontroller and haptic amplifier, which converts microcontroller’s digital output into signals that drive the actuators. The haptic actuators can be controlled by signals arriving via serial, USB, and Bluetooth Low Energy (BLE), as well as onboard microphones, using an nRF52840 microcontroller, which was chosen because it offers many input and output options and BLE, all in a small package. We added several sensors into the board to provide more experimental flexibility: an on-board digital microphone, an analog microphone amplifier, and an accelerometer. The firmware is a portable C/C++ library that works in the Arduino ecosystem.

To allow for rapid iteration during development, the interface between the board and actuators is critical. The 12 tactile signals’ wiring have to be quick to set up in order to allow for such development, while being flexible and robust to stand up to prolonged use. For the interface, we use a 24-pin FPC (flexible printed circuit) connector on the VHP. We support interfacing to the actuators in two ways: with a custom flexible circuit board and with a rigid breakout board.

VHP board (small board on the right) connected to three different types of tactile actuators via rigid breakout board (large board on the left).

Using Haptic Actuators as Sensors
In our previous blog post, we explored how back-EMF in a haptic actuator could be used for sensing and demonstrated a variety of useful applications. Instead of using back-EMF sensing in the VHP system, we measure the electrical current that drives each vibrotactile actuator and use the current load as the sensing mechanism. Unlike back-EMF sensing, this current-sensing approach allows simultaneous sensing and actuation, while minimizing the additional space needed on the board.

One challenge with the current-sensing approach is that there is a wide variety of vibrotactile actuators, each of which may behave differently and need different presets. In addition, because different actuators can be added and removed during prototyping with the adapter board, it would be useful if the VHP were able to identify the actuator automatically. This would improve the speed of prototyping and make the system more novice-friendly.

To explore this possibility, we collected current-load data from three off-the-shelf haptic actuators and trained a simple support vector machine classifier to recognize the difference in the signal pattern between actuators. The test accuracy was 100% for classifying the three actuators, indicating that each actuator has a very distinct response.

Different actuators have a different current signature during a frequency sweep, thus allowing for automatic identification.

Additionally, vibrotactile actuators require proper contact with the skin for consistent control over stimulation. Thus, the device should measure skin contact and either provide an alert or self-adjust if it is not loaded correctly. To test whether a skin contact measuring technique works in practice, we measured the current load on actuators in a bracelet as it was tightened and loosened around the wrist. As the bracelet strap is tightened, the contact pressure between the skin and the actuator increases and the current required to drive the actuator signal increases commensurately.

Current load sensing is responding to touch, while the actuator is driven at 250 Hz frequency.

Quality of the fit of the bracelet is measured.

Audio-to-Tactile Feedback
To demonstrate the utility of the VHP platform, we used it to develop an audio-to-tactile feedback device to help with lipreading. Lipreading can be difficult for many speech sounds that look similar (visemes), such as “pin” and “min”. In order to help the user differentiate visemes like these, we attach a microphone to the VHP system, which can then pick up the speech sounds and translate the audio to vibrations on the wrist. For audio-to-tactile translation, we used our previously developed algorithms for real-time audio-to-tactile conversion, available via GitHub. Briefly, audio filters are paired with neural networks to recognize certain viesemes (e.g., picking up the hard consonant “p” in “pin”), and are then translated to vibrations in different parts of the bracelet. Our approach is inspired by tactile phonemic sleeve (TAPS), however the major difference is that in our approach the tactile signal is presented continuously and in real-time.

One of the developers who employs lipreading in daily life wore the bracelet daily for several months and found it to give better information to facilitate lipreading than previous devices, allowing improved understanding of lipreading visemes with the bracelet versus lipreading alone. In the future, we plan to conduct full-scale experiments with multiple users wearing the device for an extended time.

Left: Audio-to-tactile sleeve. Middle: Audio-to-tactile bracelet. Right: One of our developers tests out the bracelets, which are worn on both arms.

Potential Applications
The VHP platform enables rapid experimentation and prototyping that can be used to develop techniques for a variety of applications. For example:

  • Rich haptics on small devices: Expanding the number of actuators on mobile phones, which typically only have one or two, could be useful to provide additional tactile information. This is especially useful as fingers are sensitive to vibrations. We demonstrated a prototype mobile phone case with eight vibrotactile actuators. This could be used to provide rich notifications and enhance effects in a mobile game or when watching a video.
  • Lab psychophysical experiments: Because VHP can be easily set up to send and receive haptic signals in real time, e.g., from a Jupyter notebook, it could be used to perform real-time haptic experiments.
  • Notifications and alerts: The wearable VHP could be used to provide haptic notifications from other devices, e.g., alerting if someone is at the door, and could even communicate distinguishable alerts through use of multiple actuators.
  • Sensory substitution: Besides the lipreading assistance example above, there are many other potential applications for accessibility using sensory substitution, such as visual-to-tactile sensing or even sensing magnetic fields.
  • Loading sensing: The ability to sense from the haptic actuator current load is unique to our platform, and enables a variety of features, such as pressure sensing or automatically adjusting actuator output.
Integrating eight voice coils into a phone case. We used loading sensing to understand which voice coils are being touched.

What's next?
We hope that others can utilize the platform to build a diverse set of applications. If you are interested and have ideas about using our platform or want to receive updates, please fill out this form. We hope that with this platform, we can help democratize the use of haptics and inspire a more widespread use of tactile devices.

This work was done by Artem Dementyev, Pascal Getreuer, Dimitri Kanevsky, Malcolm Slaney and Richard Lyon. We thank Alex Olwal, Thad Starner, Hong Tan, Charlotte Reed, Sarah Sterman for valuable feedback and discussion on the paper. Yuhui Zhao, Dmitrii Votintcev, Chet Gnegy, Whitney Bai and Sagar Savla for feedback on the design and engineering.

Source: Google AI Blog