In honor of Black History Month, Google hosted a Pay It Forward Challenge to recognize Black student leaders who are advancing opportunities for their local communities. After receiving so many submissions we’re excited to share the work of the students below and hope you’ll be inspired by their stories. Stay tuned for more features over the next few weeks! Blessing Adogame
Blessing is the co-founder of the Students of LinkedIn movement at Drexel University. This campaign started as a hashtag, and “ bloomed into a global community of students who encourage each other to find their voice, expose the limitless opportunities that they can find/create, and educate each other on the importance of personal branding as college students.”
"I wanted to create this community, because there is power in lifting others as we climb. Through my posts, I have been able to serve as an example of how a college student can provide value to industry professionals around the world and have garnered an audience of more than 5,000 students. Students of LinkedIn equips students with resources such as monthly webinars and organic video content that teaches how to connect with professionals and land/create that dream job.”
Blessing’s advice to others: “Find joy in learning from your mistakes, knowing that it leads to growth. There is power in starting, failing at something, and learning from it."
What inspires Blessing about Black History Month:
“As a community, it’s important to celebrate the sacrifices that our ancestors made. Sacrifices, that have enabled us to be able to do what we do/want to do with our lives. It’s important to be gracious towards those have made sacrifices for us - those from the past and those from the present. Growth equals success. As a community, we need to continue to grow and strengthen as we display black excellence and unity."
Natsai Ndebele
Natsai is currently a student at Georgia State University. She is the co-founder of Our Journey Through Code and a mentor with organizations such as Girls Who Code and Black Girls Code. Natsai started her non-profit, Journey Through Code with a purpose of, “increasing diversity in technology, especially for women and underrepresented minorities."
“By highlighting the many faces in technology, I hope to inspire more minorities to enter the tech world, break stereotypes and biases, and build a community for us with mentors while creating inclusion.”
Natsai’s advice to others: “The best time to start is now. You don't have to wait to get a team or gain resources or recognition, just do it. When I started, I didn't have much but I had knowledge that could be passed on. So I volunteered with organizations with missions aligned to my cause, and I reached out to people to hold events, panels, or workshops to teach girls how to code.”
What inspires Natsai about Black History Month:
“I'm really excited to celebrate being black, and highlight all the people that have contributed to the black community. There is still so much that we need to work on as a community, and so many challenges we still face, and I want this focus on building on our community to last beyond a month or just a hashtag.”
Alysa Miller
Alysa is currently pursuing her Doctoral degree in Psychology from the University of Chicago after recieving her Master’s in Public Health from New York University. Alysa has always had a passion for finding ways to improve health within underrepresented communities, including in her hometown of Detroit, MI.
"Through these educational and professional experiences I have had the opportunity to engage in a collaborative and interdisciplinary approach to learning about contributing to social change and promoting health equity. Specifically, my passion for food and improving the world’s diet, particularly in minority communities, has influenced my interest in learning the different approaches communities take toward food policy, food culture, and health disparities.
As I have made advancements on my career path, I have noticed the scarcity in strong female, minority, role models in the health sciences fields. Because of my experiences as a woman of color, I feel it is my duty to serve as a scholarly role model for future minority scholars in the health sciences field”
Alysa’s advice to others:
“Wise words I strive to live by are, 'make data driven decisions.' I would tell that person to do their share of research to make their dream a reality through decisions driven by data. Numbers carry weight and having others to bounce ideas off of and stimulate meaningful conversation goes a long way. Also when collaborating, engage the community. Educate and empower them, and strive for sustainability."
What inspires Alysa about Black History Month:
“In our current political climate, it’s especially crucial for young minorities to see minorities old and young making impactful change and positive social contributions. The internet and mainstream media have allowed all sorts of messages to come into our daily lives, and many of us over time become desensitized to the negativity and hatred spanning our world. Utilizing a platform as influential as Google to acknowledge, promote, and encourage commitment to community serves as a reminder of what Black History Month (and those who fought for equality) embodies – particularly at a time when diversity and inclusion are needed most.”
Rayna Dunham
Rayna is both a student and a teacher. She is a high school biology teacher and a member of the Greater New England Alliance of Black School Educators while she simultaneously attends Central Connecticut State University. On top of that she volunteers at the Legacy Foundation of Hartford, a local food pantry.
“Upon completing an internship as a public health educator at the age of nineteen, I made the decision to become a high school biology teacher in an urban setting. I specifically wanted to work in an urban setting to serve as a role model for my students impacted by the vision gap disallowing them the privilege of being educated by a black teacher in a core subject area. I quickly learned that being a black teacher is more than just teaching the curriculum. I dedicate myself to each scholar’s academic success by being present every week and instilling a love of science into every potential doctor and engineer I teach.”
Rayna’s advice to others: “Become a member of an existing organization that aligns with your goals and devote your time to improving it or expanding it with your resources. The most valuable gift you can give someone is your time.”
What inspires Rayna about Black History Month: “I will use my postgraduate education to create generational change by disseminating my research in the community at large and investing in future teachers. Black History Month makes me reflect on the current realities of the world and inspires me to combat them in my classroom by teaching and furthering my own education. I will be the change this world needs by any means necessary." Saba Tshibaka
Saba is currently a student at the University of Maryland, College Park. On and off campus, Saba participates in multiple organizations. She is the founder of the Jeopardy! Club, a computer literacy tutor in BcauseIcan, part of the University of Maryland Black Student Union's Big/Little Mentoring Program, and an ambassador for the Academic Achievement Program.
“At the Jeopardy! Club meetings, we invite not only students, but staff, faculty, and most importantly neighborhood/community members. It brings me great joy to be the president of an organization whose mission statement is to, 'commemorate the wealth of ALL of our knowledge!'"
Saba’s advice to others:
“My advice to someone looking to make an impact in their local community is focus on what you're passionate about! I've been watching Jeopardy since I was 7 with my family, and one of my biggest dreams was to act like Alex Trebek in Jeopardy, and now I get to do that monthly in a group run by my closest friends! Anything is possible if you try your hardest and believe in your work. Having a positive mindset makes the biggest difference.”
What inspires Saba about Black History Month:
“Something on my mind as we enter Black History month is how to better combat stereotypes against minority groups. So far I've just been trying to work as hard as I can to be the opposite of what people think I am. As a young black woman, I've gone to college, held countless jobs, and work to support my family as well as myself while staying hopeful."
Helping Android app developers build secure apps, free of known vulnerabilities, means helping the overall ecosystem thrive. This is why we launched the Application Security Improvement Program five years ago, and why we're still so invested in its success today.
What the Android Security Improvement Program does
When an app is submitted to the Google Play store, we scan it to determine if a variety of vulnerabilities are present. If we find something concerning, we flag it to the developer and then help them to remedy the situation.
Think of it like a routine physical. If there are no problems, the app runs through our normal tests and continues on the process to being published in the Play Store. If there is a problem, however, we provide a diagnosis and next steps to get back to healthy form.
Over its lifetime, the program has helped more than 300,000 developers to fix more than 1,000,000 apps on Google Play. In 2018 alone, the program helped over 30,000 developers fix over 75,000 apps. The downstream effect means that those 75,000 vulnerable apps are not distributed to users with the same security issues present, which we consider a win.
What vulnerabilities are covered
The App Security Improvement program covers a broad range of security issues in Android apps. These can be as specific as security issues in certain versions of popular libraries (ex: CVE-2015-5256) and as broad as unsafe TLS/SSL certificate validation.
We are continuously improving this program's capabilities by improving the existing checks and launching checks for more classes of security vulnerability. In 2018, we deployed warnings for six additional security vulnerability classes including:
SQL Injection
File-based Cross-Site Scripting
Cross-App Scripting
Leaked Third-Party Credentials
Scheme Hijacking
JavaScript Interface Injection
Ensuring that we're continuing to evolve the program as new exploits emerge is a top priority for us. We are continuing to work on this throughout 2019.
Keeping Android users safe is important to Google. We know that app security is often tricky and that developers can make mistakes. We hope to see this program grow in the years to come, helping developers worldwide build apps users can truly trust.
Posted by Patrick Mutchler and Meghan Kelly, Android Security & Privacy Team
Helping Android app developers build secure apps, free of known vulnerabilities, means helping the overall ecosystem thrive. This is why we launched the Application Security Improvement Program five years ago, and why we're still so invested in its success today.
What the Android Security Improvement Program does
When an app is submitted to the Google Play store, we scan it to determine if a variety of vulnerabilities are present. If we find something concerning, we flag it to the developer and then help them to remedy the situation.
Think of it like a routine physical. If there are no problems, the app runs through our normal tests and continues on the process to being published in the Play Store. If there is a problem, however, we provide a diagnosis and next steps to get back to healthy form.
Over its lifetime, the program has helped more than 300,000 developers to fix more than 1,000,000 apps on Google Play. In 2018 alone, the program helped over 30,000 developers fix over 75,000 apps. The downstream effect means that those 75,000 vulnerable apps are not distributed to users with the same security issues present, which we consider a win.
What vulnerabilities are covered
The App Security Improvement program covers a broad range of security issues in Android apps. These can be as specific as security issues in certain versions of popular libraries (ex: CVE-2015-5256) and as broad as unsafe TLS/SSL certificate validation.
We are continuously improving this program's capabilities by improving the existing checks and launching checks for more classes of security vulnerability. In 2018, we deployed warnings for six additional security vulnerability classes including:
SQL Injection
File-based Cross-Site Scripting
Cross-App Scripting
Leaked Third-Party Credentials
Scheme Hijacking
JavaScript Interface Injection
Ensuring that we're continuing to evolve the program as new exploits emerge is a top priority for us. We are continuing to work on this throughout 2019.
Keeping Android users safe is important to Google. We know that app security is often tricky and that developers can make mistakes. We hope to see this program grow in the years to come, helping developers worldwide build apps users can truly trust.
We know that many of you have been closely following the actions we’re taking to protect young people on YouTube and are as deeply concerned as we are that we get this right. We want to update you on some additional changes we’re making, particularly in regards to comments, building on the efforts we shared last week.
We recognize that comments are a core part of the YouTube experience and how you connect with and grow your audience. At the same time, the important steps we’re sharing today are critical for keeping young people safe. Thank you for your understanding and feedback as we continue our work to protect the YouTube community.
Below is a summary of the main steps we’ve taken to improve child safety on YouTube since our update last Friday:
Disabling comments on videos featuring minors
Over the past week, we disabled comments from tens of millions of videos that could be subject to predatory behavior. These efforts are focused on videos featuring young minors and we will continue to identify videos at risk over the next few months. Over the next few months, we will be broadening this action to suspend comments on videos featuring young minors and videos featuring older minors that could be at risk of attracting predatory behavior.
A small number of creators will be able to keep comments enabled on these types of videos. These channels will be required to actively moderate their comments, beyond just using our moderation tools, and demonstrate a low risk of predatory behavior. We will work with them directly and our goal is to grow this number over time as our ability to catch violative comments continues to improve.
Launching a new comments classifier
While we have been removing hundreds of millions of comments for violating our policies, we had been working on an even more effective classifier, that will identify and remove predatory comments. This classifier does not affect the monetization of your video. We accelerated its launch and now have a new comments classifier in place that is more sweeping in scope, and will detect and remove 2X more individual comments.
Taking action on creators who cause egregious harm to the community
No form of content that endangers minors is acceptable on YouTube, which is why we have terminated certain channels that attempt to endanger children in any way. Videos encouraging harmful and dangerous challenges targeting any audience are also clearly against our policies. We will continue to take action when creators violate our policies in ways that blatantly harm the broader user and creator community. Please continue to flag these to us.
Thank you for your understanding as we make these changes,
My mother was the first African American bank teller in my hometown in North Carolina in the 1960s. As you can imagine, this came with a lot of challenges. Team lunches and office holiday parties were sometimes hosted at racially segregated restaurants where they wouldn't serve my mother, and customers refused to work with her because of the color of her skin. Though these experiences were disconcerting for her, she stayed at the bank for more than 20 years where she progressed her career, helped transition the bank to a more inclusive workplace and paved the way for other women to take on similar roles.
Her story left an imprint on me. While she persevered through these challenges, she could have just as easily left. How would that have affected her career, the culture of the bank and its community and the experiences of people who worked there after her? How would her experience have been different if she wasn’t the only one? These questions fuel my work to address diversity, equity and inclusion in the workplace.
Measuring up the challenges
Lack of diversity in corporate America is a well-documented problem and improvements have been slow. I’ve seen this first hand throughout my career: from serving as a champion for the Department of Energy’s Minorities in Energy Initiative and leading diversity programs at Lockheed Martin to my current role where I lead our diversity, equity and inclusion efforts at Google.
I’ve learned over the years that you can’t fix what you don’t measure. Representation is a function of many factors such as hiring, development, progression, retention and culture. Without measuring these things, it is hard to know what changes need to be made. Which is why, at Google, we’ve been taking a second look at what and who we measure.
Specifically, we started paying closer attention to attrition rates so we know how many employees stay and leave our company each year. Why? Because we can’t improve representation without knowing which employees are leaving faster than others. More importantly, there are human beings behind the numbers we report each year, and how well we retain and develop talent has a real impact on people's careers and lives. That’s why we care about getting this right, and getting better.
Last year, Google published its first attrition index, and the results were mixed. Attrition rates indicate how many employees leave a company annually. Globally, women were leaving Google at lower rates than the average, and in the U.S., where we are able to report across race, Black and Latinx Googlers were leaving at faster rates than the average. Looking at these numbers, I always think about my mother’s experience at the bank. What would they have lost if they had failed to keep her there? Would they have been able to fix the roadblocks that ended her career? Thankfully, by measuring these numbers at Google, we can implement initiatives across Google to find solutions. Here’s a look at the improvements and progress we have made over the past year, and the work we have ahead.
Finding the gaps and mending them
To improve attrition, it’s our job to make sure underrepresented employees find satisfaction in their role, feel included at work and have opportunities to develop and grow.
It can be difficult to know where to go for career support, especially when you’re talking about a company as big as Google. This can be even more challenging for underrepresented groups who might not have representation in leadership, sponsorship or an existing network to rely on. To combat this, we created a new team of Retention Case Managers that I help lead. Retention Case Managers help connect employees with the right resources, whether it’s training for a new role or providing a safe space to share concerns and find community. We’ve already seen the positive impact this has on attrition, and look forward to expanding this program more broadly across the company.
We’re also always looking for ways to build and support community across the company. For example, my team created a State of Black Women event at Google where we brought together more than 500 black female employees from across the globe to meet with Google’s CEO, VP of Employee Engagement and myself. This not only created a space for the community to come together, but also allowed us as leaders to get closer to the challenges that our employees face.
We've learned that fostering community from the very beginning is important. Over the past year, we worked with internal Employee Resource Groups to make sure Black and Latinx Googlers have access to a community and a mentor to help navigate the beginning of their careers at Google. To date, nearly 200 Black and Latinx new employees have been paired with mentors across three countries. In 2019, we will be expanding this partnership to more Employee Resource Groups that support women and employees with disabilities.
Improvements in attrition year over year
A year after implementing these changes, 2018 attrition rates have improved for the majority of demographic groups across the company. Specially, Black and Latinx Googlers in the U.S. saw some of the largest year over year improvements in attrition.
While there are positive trends, there is still work to be done. Specifically, attrition for Native Americans worsened. And while rates improved for Black and Latinx Googlers, they are still not on par with the average. These are all areas we plan to focus on over the coming year.
Diversity in the workplace is a hard problem to solve. By identifying challenges that we face across hiring, development, progression and attrition, we can understand where we should focus our efforts and affect real change to build a diverse and inclusive workplace for everyone.
My mother was the first African American bank teller in my hometown in North Carolina in the 1960s. As you can imagine, this came with a lot of challenges. Team lunches and office holiday parties were sometimes hosted at racially segregated restaurants where they wouldn't serve my mother, and customers refused to work with her because of the color of her skin. Though these experiences were disconcerting for her, she stayed at the bank for more than 20 years where she progressed her career, helped transition the bank to a more inclusive workplace and paved the way for other women to take on similar roles.
Her story left an imprint on me. While she persevered through these challenges, she could have just as easily left. How would that have affected her career, the culture of the bank and its community and the experiences of people who worked there after her? How would her experience have been different if she wasn’t the only one? These questions fuel my work to address diversity, equity and inclusion in the workplace.
Measuring up the challenges
Lack of diversity in corporate America is a well-documented problem and improvements have been slow. I’ve seen this first hand throughout my career: from serving as a champion for the Department of Energy’s Minorities in Energy Initiative and leading diversity programs at Lockheed Martin to my current role where I lead our diversity, equity and inclusion efforts at Google.
I’ve learned over the years that you can’t fix what you don’t measure. Representation is a function of many factors such as hiring, development, progression, retention and culture. Without measuring these things, it is hard to know what changes need to be made. Which is why, at Google, we’ve been taking a second look at what and who we measure.
Specifically, we started paying closer attention to attrition rates so we know how many employees stay and leave our company each year. Why? Because we can’t improve representation without knowing which employees are leaving faster than others. More importantly, there are human beings behind the numbers we report each year, and how well we retain and develop talent has a real impact on people's careers and lives. That’s why we care about getting this right, and getting better.
Last year, Google published its first attrition index, and the results were mixed. Attrition rates indicate how many employees leave a company annually. Globally, women were leaving Google at lower rates than the average, and in the U.S., where we are able to report across race, Black and Latinx Googlers were leaving at faster rates than the average. Looking at these numbers, I always think about my mother’s experience at the bank. What would they have lost if they had failed to keep her there? Would they have been able to fix the roadblocks that ended her career? Thankfully, by measuring these numbers at Google, we can implement initiatives across Google to find solutions. Here’s a look at the improvements and progress we have made over the past year, and the work we have ahead.
Finding the gaps and mending them
To improve attrition, it’s our job to make sure underrepresented employees find satisfaction in their role, feel included at work and have opportunities to develop and grow.
It can be difficult to know where to go for career support, especially when you’re talking about a company as big as Google. This can be even more challenging for underrepresented groups who might not have representation in leadership, sponsorship or an existing network to rely on. To combat this, we created a new team of Retention Case Managers that I help lead. Retention Case Managers help connect employees with the right resources, whether it’s training for a new role or providing a safe space to share concerns and find community. We’ve already seen the positive impact this has on attrition, and look forward to expanding this program more broadly across the company.
We’re also always looking for ways to build and support community across the company. For example, my team created a State of Black Women event at Google where we brought together more than 500 black female employees from across the globe to meet with Google’s CEO, VP of Employee Engagement and myself. This not only created a space for the community to come together, but also allowed us as leaders to get closer to the challenges that our employees face.
We've learned that fostering community from the very beginning is important. Over the past year, we worked with internal Employee Resource Groups to make sure Black and Latinx Googlers have access to a community and a mentor to help navigate the beginning of their careers at Google. To date, nearly 200 Black and Latinx new employees have been paired with mentors across three countries. In 2019, we will be expanding this partnership to more Employee Resource Groups that support women and employees with disabilities.
Improvements in attrition year over year
A year after implementing these changes, 2018 attrition rates have improved for the majority of demographic groups across the company. Specially, Black and Latinx Googlers in the U.S. saw some of the largest year over year improvements in attrition.
While there are positive trends, there is still work to be done. Specifically, attrition for Native Americans worsened. And while rates improved for Black and Latinx Googlers, they are still not on par with the average. These are all areas we plan to focus on over the coming year.
Diversity in the workplace is a hard problem to solve. By identifying challenges that we face across hiring, development, progression and attrition, we can understand where we should focus our efforts and affect real change to build a diverse and inclusive workplace for everyone.
Aleksandra Faust, Senior Research Scientist and Anthony Francis, Senior Software Engineer, Robotics at Google
In the United States alone, there are 3 million people with a mobility impairment that prevents them from ever leaving their homes. Service robots that can autonomously navigate long distances can improve the independence of people with limited mobility, for example, by bringing them groceries, medicine, and packages. Research has demonstrated that deep reinforcement learning (RL) is good at mapping raw sensory input to actions, e.g. learning to grasp objects and for robot locomotion, but RL agents usually lack the understanding of large physical spaces needed to safely navigate long distances without human help and to easily adapt to new spaces.
In three recent papers, “Learning Navigation Behaviors End-to-End with AutoRL,” “PRM-RL: Long-Range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning”, and “Long-Range Indoor Navigation with PRM-RL”, we investigate easy-to-adapt robotic autonomy by combining deep RL with long-range planning. We train local planner agents to perform basic navigation behaviors, traversing short distances safely without collisions with moving obstacles. The local planners take noisy sensor observations, such as a 1D lidar that provides distances to obstacles, and output linear and angular velocities for robot control. We train the local planner in simulation with AutoRL, a method that automates the search for RL reward and neural network architecture. Despite their limited range of 10 - 15 meters, the local planners transfer well to both real robots and to new, previously unseen environments. This enables us to use them as building blocks for navigation in large spaces. We then build a roadmap, a graph where nodes are locations and edges connect the nodes only if local planners, which mimic real robots well with their noisy sensors and control, can traverse between them reliably.
Automating Reinforcement Learning (AutoRL) In our first paper, we train the local planners in small, static environments. However, training with standard deep RL algorithms, such as Deep Deterministic Policy Gradient (DDPG), poses several challenges. For example, the true objective of the local planners is to reach the goal, which represents a sparse reward. In practice, this requires researchers to spend significant time iterating and hand-tuning the rewards. Researchers must also make decisions about the neural network architecture, without clear accepted best practices. And finally, algorithms like DDPG are unstable learners and often exhibit catastrophic forgetfulness.
To overcome those challenges, we automate the deep Reinforcement Learning (RL) training. AutoRL is an evolutionary automation layer around deep RL that searches for a reward and neural network architecture using large-scale hyperparameter optimization. It works in two phases, reward search and neural network architecture search. During the reward search, AutoRL trains a population of DDPG agents concurrently over several generations, each with a slightly different reward function optimizing for the local planner’s true objective: reaching the destination. At the end of the reward search phase, we select the reward that leads the agents to its destination most often. In the neural network architecture search phase, we repeat the process, this time using the selected reward and tuning the network layers, optimizing for the cumulative reward.
Automating reinforcement learning with reward and neural network architecture search.
However, this iterative process means AutoRL is not sample efficient. Training one agent takes 5 million samples; AutoRL training over 10 generations of 100 agents requires 5 billion samples - equivalent to 32 years of training! The benefit is that after AutoRL the manual training process is automated, and DDPG does not experience catastrophic forgetfulness. Most importantly, the resulting policies are higher quality — AutoRL policies are robust to sensor, actuator and localization noise, and generalize well to new environments. Our best policy is 26% more successful than other navigation methods across our test environments.
AutoRL local planner policy transfer to robots in real, unstructured environments
While these policies only perform local navigation, they are robust to moving obstacles and transfer well to real robots, even in unstructured environments. Though they were trained in simulation with only static obstacles, they can also handle moving objects effectively. The next step is to combine the AutoRL policies with sampling-based planning to extend their reach and enable long-range navigation.
Achieving Long Range Navigation with PRM-RL Sampling-based planners tackle long-range navigation by approximating robot motions. For example, probabilistic roadmaps (PRMs) sample robot poses and connect them with feasible transitions, creating roadmaps that capture valid movements of a robot across large spaces. In our second paper, which won Best Paper in Service Robotics at ICRA 2018, we combine PRMs with hand-tuned RL-based local planners (without AutoRL) to train robots once locally and then adapt them to different environments.
First, for each robot we train a local planner policy in a generic simulated training environment. Next, we build a PRM with respect to that policy, called a PRM-RL, over a floor plan for the deployment environment. The same floor plan can be used for any robot we wish to deploy in the building in a one time per robot+environment setup.
To build a PRM-RL we connect sampled nodes only if the RL-based local planner, which represents robot noise well, can reliably and consistently navigate between them. This is done via Monte Carlo simulation. The resulting roadmap is tuned to both the abilities and geometry of the particular robot. Roadmaps for robots with the same geometry but different sensors and actuators will have different connectivity. Since the agent can navigate around corners, nodes without clear line of sight can be included. Whereas nodes near walls and obstacles are less likely to be connected into the roadmap because of sensor noise. At execution time, the RL agent navigates from roadmap waypoint to waypoint.
Roadmap being built with 3 Monte Carlo simulations per randomly selected node pair.
The largest map was 288 meters by 163 meters and contains almost 700,000 edges, collected over 4 days using 300 workers in a cluster requiring 1.1 billion collision checks.
The third paper makes several improvements over the original PRM-RL. First, we replace the hand-tuned DDPG with AutoRL-trained local planners, which results in improved long-range navigation. Second, it adds Simultaneous Localization and Mapping (SLAM) maps, which robots use at execution time, as a source for building the roadmaps. Because SLAM maps are noisy, this change closes the “sim2real gap”, a phonomena in robotics where simulation-trained agents significantly underperform when transferred to real-robots. Our simulated success rates are the same as in on-robot experiments. Last, we added distributed roadmap building, resulting in very large scale roadmaps containing up to 700,000 nodes.
We evaluated the method using our AutoRL agent, building roadmaps using the floor maps of offices up to 200x larger than the training environments, accepting edges with at least 90% success over 20 trials. We compared PRM-RL to a variety of different methods over distances up to 100m, well beyond the local planner range. PRM-RL had 2 to 3 times the rate of success over baseline because the nodes were connected appropriately for the robot’s capabilities.
We tested PRM-RL on multiple real robots and real building sites. One set of tests are shown below; the robot is very robust except near cluttered areas and off the edge of the SLAM map.
On-robot experiments
Conclusion Autonomous robot navigation can significantly improve independence of people with limited mobility. We can achieve this by development of easy-to-adapt robotic autonomy, including methods that can be deployed in new environments using information that it is already available. This is done by automating the learning of basic, short-range navigation behaviors with AutoRL and using these learned policies in conjunction with SLAM maps to build roadmaps. These roadmaps consist of nodes connected by edges that robots can traverse consistently. The result is a policy that once trained can be used across different environments and can produce a roadmap custom-tailored to the particular robot.
Acknowledgements The research was done by, in alphabetical order, Hao-Tien Lewis Chiang, James Davidson, Aleksandra Faust, Marek Fiser, Anthony Francis, Jasmine Hsu, J. Chase Kew, Tsang-Wei Edward Lee, Ken Oslund, Oscar Ramirez from Robotics at Google and Lydia Tapia from University of New Mexico. We thank Alexander Toshev, Brian Ichter, Chris Harris, and Vincent Vanhoucke for helpful discussions.
Aleksandra Faust, Senior Research Scientist and Anthony Francis, Senior Software Engineer, Robotics at Google
In the United States alone, there are 3 million people with a mobility impairment that prevents them from ever leaving their homes. Service robots that can autonomously navigate long distances can improve the independence of people with limited mobility, for example, by bringing them groceries, medicine, and packages. Research has demonstrated that deep reinforcement learning (RL) is good at mapping raw sensory input to actions, e.g. learning to grasp objects and for robot locomotion, but RL agents usually lack the understanding of large physical spaces needed to safely navigate long distances without human help and to easily adapt to new spaces.
In three recent papers, “Learning Navigation Behaviors End-to-End with AutoRL,” “PRM-RL: Long-Range Robotic Navigation Tasks by Combining Reinforcement Learning and Sampling-based Planning”, and “Long-Range Indoor Navigation with PRM-RL”, we investigate easy-to-adapt robotic autonomy by combining deep RL with long-range planning. We train local planner agents to perform basic navigation behaviors, traversing short distances safely without collisions with moving obstacles. The local planners take noisy sensor observations, such as a 1D lidar that provides distances to obstacles, and output linear and angular velocities for robot control. We train the local planner in simulation with AutoRL, a method that automates the search for RL reward and neural network architecture. Despite their limited range of 10 - 15 meters, the local planners transfer well to both real robots and to new, previously unseen environments. This enables us to use them as building blocks for navigation in large spaces. We then build a roadmap, a graph where nodes are locations and edges connect the nodes only if local planners, which mimic real robots well with their noisy sensors and control, can traverse between them reliably.
Automating Reinforcement Learning (AutoRL) In our first paper, we train the local planners in small, static environments. However, training with standard deep RL algorithms, such as Deep Deterministic Policy Gradient (DDPG), poses several challenges. For example, the true objective of the local planners is to reach the goal, which represents a sparse reward. In practice, this requires researchers to spend significant time iterating and hand-tuning the rewards. Researchers must also make decisions about the neural network architecture, without clear accepted best practices. And finally, algorithms like DDPG are unstable learners and often exhibit catastrophic forgetfulness.
To overcome those challenges, we automate the deep Reinforcement Learning (RL) training. AutoRL is an evolutionary automation layer around deep RL that searches for a reward and neural network architecture using large-scale hyperparameter optimization. It works in two phases, reward search and neural network architecture search. During the reward search, AutoRL trains a population of DDPG agents concurrently over several generations, each with a slightly different reward function optimizing for the local planner’s true objective: reaching the destination. At the end of the reward search phase, we select the reward that leads the agents to its destination most often. In the neural network architecture search phase, we repeat the process, this time using the selected reward and tuning the network layers, optimizing for the cumulative reward.
Automating reinforcement learning with reward and neural network architecture search.
However, this iterative process means AutoRL is not sample efficient. Training one agent takes 5 million samples; AutoRL training over 10 generations of 100 agents requires 5 billion samples - equivalent to 32 years of training! The benefit is that after AutoRL the manual training process is automated, and DDPG does not experience catastrophic forgetfulness. Most importantly, the resulting policies are higher quality — AutoRL policies are robust to sensor, actuator and localization noise, and generalize well to new environments. Our best policy is 26% more successful than other navigation methods across our test environments.
AutoRL local planner policy transfer to robots in real, unstructured environments
While these policies only perform local navigation, they are robust to moving obstacles and transfer well to real robots, even in unstructured environments. Though they were trained in simulation with only static obstacles, they can also handle moving objects effectively. The next step is to combine the AutoRL policies with sampling-based planning to extend their reach and enable long-range navigation.
Achieving Long Range Navigation with PRM-RL Sampling-based planners tackle long-range navigation by approximating robot motions. For example, probabilistic roadmaps (PRMs) sample robot poses and connect them with feasible transitions, creating roadmaps that capture valid movements of a robot across large spaces. In our second paper, which won Best Paper in Service Robotics at ICRA 2018, we combine PRMs with hand-tuned RL-based local planners (without AutoRL) to train robots once locally and then adapt them to different environments.
First, for each robot we train a local planner policy in a generic simulated training environment. Next, we build a PRM with respect to that policy, called a PRM-RL, over a floor plan for the deployment environment. The same floor plan can be used for any robot we wish to deploy in the building in a one time per robot+environment setup.
To build a PRM-RL we connect sampled nodes only if the RL-based local planner, which represents robot noise well, can reliably and consistently navigate between them. This is done via Monte Carlo simulation. The resulting roadmap is tuned to both the abilities and geometry of the particular robot. Roadmaps for robots with the same geometry but different sensors and actuators will have different connectivity. Since the agent can navigate around corners, nodes without clear line of sight can be included. Whereas nodes near walls and obstacles are less likely to be connected into the roadmap because of sensor noise. At execution time, the RL agent navigates from roadmap waypoint to waypoint.
Roadmap being built with 3 Monte Carlo simulations per randomly selected node pair.
The largest map was 288 meters by 163 meters and contains almost 700,000 edges, collected over 4 days using 300 workers in a cluster requiring 1.1 billion collision checks.
The third paper makes several improvements over the original PRM-RL. First, we replace the hand-tuned DDPG with AutoRL-trained local planners, which results in improved long-range navigation. Second, it adds Simultaneous Localization and Mapping (SLAM) maps, which robots use at execution time, as a source for building the roadmaps. Because SLAM maps are noisy, this change closes the “sim2real gap”, a phonomena in robotics where simulation-trained agents significantly underperform when transferred to real-robots. Our simulated success rates are the same as in on-robot experiments. Last, we added distributed roadmap building, resulting in very large scale roadmaps containing up to 700,000 nodes.
We evaluated the method using our AutoRL agent, building roadmaps using the floor maps of offices up to 200x larger than the training environments, accepting edges with at least 90% success over 20 trials. We compared PRM-RL to a variety of different methods over distances up to 100m, well beyond the local planner range. PRM-RL had 2 to 3 times the rate of success over baseline because the nodes were connected appropriately for the robot’s capabilities.
We tested PRM-RL on multiple real robots and real building sites. One set of tests are shown below; the robot is very robust except near cluttered areas and off the edge of the SLAM map.
On-robot experiments
Conclusion Autonomous robot navigation can significantly improve independence of people with limited mobility. We can achieve this by development of easy-to-adapt robotic autonomy, including methods that can be deployed in new environments using information that it is already available. This is done by automating the learning of basic, short-range navigation behaviors with AutoRL and using these learned policies in conjunction with SLAM maps to build roadmaps. These roadmaps consist of nodes connected by edges that robots can traverse consistently. The result is a policy that once trained can be used across different environments and can produce a roadmap custom-tailored to the particular robot.
Acknowledgements The research was done by, in alphabetical order, Hao-Tien Lewis Chiang, James Davidson, Aleksandra Faust, Marek Fiser, Anthony Francis, Jasmine Hsu, J. Chase Kew, Tsang-Wei Edward Lee, Ken Oslund, Oscar Ramirez from Robotics at Google and Lydia Tapia from University of New Mexico. We thank Alexander Toshev, Brian Ichter, Chris Harris, and Vincent Vanhoucke for helpful discussions.
The Beta channel has been updated to 73.0.3683.57 (Platform version: 11647.61.0) for most Chrome OS devices. This build contains a number of bug fixes, security updates and feature enhancements. A list of changes can be found here.
If you find new issues, please let us know by visiting our forum or filing a bug. Interested in switching channels? Find out how. You can submit feedback using ‘Report an issue...’ in the Chrome menu (3 vertical dots in the upper right corner of the browser).