Category Archives: Open Source Blog

News about Google’s open source projects and programs

Introducing the Continuous Delivery Foundation, the new home for Tekton, Jenkins, Jenkins X and Spinnaker

We're excited to announce that Google is a founding member of the newly formed Continuous Delivery Foundation (CDF). Continuous delivery (CD) is a critical part of modern software development and DevOps practices, and we're excited to collaborate in a vendor-neutral foundation with other industry leaders.

We're also thrilled to announce the contribution of two projects as part of our membership: Tekton, and in collaboration with Netflix, Spinnaker. These donations will enter alongside Jenkins and Jenkins X, providing an exciting portfolio of projects for the CDF to expand upon.

Continuous Delivery Foundation

Currently, the continuous integration/continuous delivery (CI/CD) tool landscape is highly fragmented. As companies migrate to the cloud and modernize their infrastructure, tooling decisions become increasingly complicated and difficult. DevOps practitioners constantly seek guidance on software delivery best practices and how to secure their software supply chains but gathering this information can be difficult. Enter the CDF.

The CDF is about more than just code. Modern application development brings new challenges around security and compliance. This foundation will work to define the practices and guidelines that, together with tooling, will help application developers everywhere deliver better and more secure software at speed.

At a foundation level, the CDF will help make CI/CD tooling easier. And at a project level, Tekton helps address complexity problems at their core. We will team up with the open source community and industry leaders to design and build the critical pieces common to CI/CD systems.

Tekton

Tekton is a set of shared, open source components for building CI/CD systems. It provides a flexible, extensible workflow that accommodates deployment to Kubernetes, VMs, bare metal, mobile or even emerging use cases.

The project’s goal is to provide industry specifications for pipelines, workflows, source code access and other primitives. It modernizes the continuous delivery control plane by leveraging all of the built-in scaling, reliability, and extensibility advantages of Kubernetes, and moves software deployment logic there. Tekton was initially built as a part of Knative, but given its stand-alone power, and ability to deploy to a variety of targets, we’ve decided to separate its functionality out into a new project.

Today, Tekton includes primitives for pipeline definition, source code access, artifact management, and test execution. The project roadmap includes adding support for results and event triggering in the coming months. We also plan to work with CI/CD vendors to build out an ecosystem of components that will allow you to use Tekton with existing tools like Jenkins X, Knative and others.

Spinnaker

Spinnaker is an open source, multi-cloud continuous delivery platform originally created by Netflix and jointly led by Netflix and Google. It is typically used in organizations at scale, where DevOps teams support multiple development teams, and has been battle-tested in production by hundreds of teams and in millions of deployments.

Spinnaker is a multi-component system that conceptually aligns with Tekton, and that includes many features important to making continuous delivery reliable, including support for advanced deployment strategies, and Kayenta, an open source canary analysis service.

Given Google’s significant contributions to both Tekton and Spinnaker, we’re very pleased to see them become part of the same foundation. Spinnaker’s large user community has a great deal of experience in the continuous delivery domain, and joining the CDF provides a great opportunity to share that expertise with the broader community.

Next Steps

To learn more about the CDF, listen to this week's Kubernetes Podcast from Google, where the guest is Tracy Miranda, Director of Open Source Community from our partner CloudBees.

If you'd like to participate in the future of Tekton, Spinnaker, or the CDF, please join us in Barcelona, Spain, on May 20th at the Continuous Delivery Summit ahead of KubeCon/CloudNativeCon EU. If you can’t make it, don’t worry, as there will be many opportunities to get involved and become a part of the community.

We look forward to working with the continuous delivery community on shaping the next wave of CI/CD innovations, alignments, and improvements, no matter where your applications are delivered to.

By Dan Lorenc and Kim Lewandowski, DevOps at Google Cloud

Introducing Season of Docs

Google Open Source is delighted to announce Season of Docs, a new program which fosters the open source contributions of technical writers.

Season of Docs brings technical writers and open source projects together for a few months to work on open source documentation. 2019 is the first time we’re running this exciting new program.

Join us in making a substantive contribution to open source software development around the world.

Fostering collaboration between open source projects and technical writers

The Open Source Survey showed that documentation is highly valued in open source communities, yet there’s little good documentation out there. Why? Because creating documentation is hard. But...

There are people who know how to do docs well. Technical writers know how to structure a documentation site so that people can find and understand the content. They know how to write docs that fit the needs of their audience. Technical writers can also help optimize a community’s processes for open source contribution and on-boarding new contributors.

During Season of Docs, technical writers will spend a few months working closely with open source communities. Each writer works with their chosen open source project. The writers bring their expertise to the projects’ documentation while at the same time learning about open source and new technologies.

Mentors from participating open source organizations share knowledge of their communities’ processes and tools. Together the technical writers and mentors build a new doc set, improve the structure of the existing docs, develop a much-needed tutorial, or improve contribution processes and guides. See more ideas for technical writing projects.

By working together in Season of Docs we raise awareness of open source, of docs, and of technical writing.

How does it work?

  • April 2-23: Open source organizations apply to take part in Season of Docs
  • April 30: Google publishes the list of accepted mentoring organizations, along with their ideas for documentation projects
  • April 30 - June 28: Technical writers choose the project they’d like to work on and submit their proposals to Season of Docs 
  • July 30: Google announces the accepted technical writer projects
  • August 1 - September 1: Community bonding: Technical writers get to know mentors and the open source community, and refine their projects in collaboration with their mentors
  • September 2 - November 29: Technical writers work with open source mentors on the accepted projects, and submit their work at the end of the period
  • December 10: Google publishes the list of  successfully-completed projects.
See the timeline for details, including the provision for projects that run longer than three months.

Join us

Explore the Season of Docs website at g.co/seasonofdocs to learn more about participating in the program. Use our logo and other promotional resources to spread the word. Examine the timeline, check out the FAQ, and get ready to apply!

By Sarah Maddox, Google Technical Writer and Andrew Chen, Google Open Source

And the Google Summer of Code 2019 mentor organizations are…

We are excited to announce the open source projects and organizations that have been accepted for Google Summer of Code (GSoC) 2019, the 15th year of the program! As usual, we received more applications this year than we did last year, about twice as many as we are able to accept into the program.

After careful review, we have chosen 207 open source projects to be mentor organizations this year, 28 of which are new to the program. Please see the program website for a complete list of the accepted organizations.

Are you a student interested in participating? We begin accepting student applications on Monday, March 25, 2019 at 18:00 UTC and the deadline to apply is Tuesday, April 9, 2019 at 18:00 UTC.

The most successful applications come from students who start preparing now. You can start by watching the short video below, checking out the Student Guide, and reviewing the list of accepted organizations and reaching out to the 2 or 3 that interest you the most now - before the application period begins.


You can find more information on our website, including a full timeline of important dates. We also highly recommend perusing the FAQ and Program Rules and watching the short videos with more details about GSoC for students and mentors.

A hearty congratulations–and thank you–to all of our mentor organizations! We look forward to working with all of you during Google Summer of Code 2019.

By Stephanie Taylor, Google Open Source

arXiv LaTeX cleaner: safer and easier open source research papers

Open source is usually associated with code behind utilities and applications, though you can find it in many other places: such as the LaTeX source code that describes the PDFs of scientific papers.

As an example, the following source code:


Generates this PDF when compiled using pdflatex:
You can see a huge repository of such open source code at arXiv.org, an open access repository of scientific papers currently containing about 1.5 million entries (140,616 uploads in 2018). One can not only download all papers in PDF format, but also obtain the source code to regenerate them and freely reuse any of their parts.

Open sourcing LaTeX code, however, comes with its risks and challenges. We’ve built and released the code of arXiv LaTeX cleaner to remedy some of these.

Scrubbing the Code

The main risk one faces when sharing LaTeX code with the world is accidentally releasing private information, primarily through commented code left over in the file itself.

While authors put a lot of effort into polishing the final PDF, the code isn’t usually cleaned up and is left with many pieces of text that don’t actually appear in the PDF. Things like, “I do not see why the following statement should be correct,” or “Look, I’m citing you!,” make it into arXiv for everyone to see. This happens so often there’s even a Twitter bot that finds and publishes them!

Cleaning up this commented out code manually is laborious, so arXiv LaTeX cleaner automatically removes it for you.

Private information can also be found in the many auxiliary files that LaTeX generates when the code is compiled. Some of them are needed in arXiv (e.g., .bbl files), some of them are not: arXiv LaTeX cleaner will delete the unneeded ones and keep the rest automatically.

Cleaning and Autoscaling Images

Challenges also come our way when preparing the code to submit to arXiv: one needs to upload a ZIP file smaller than 10 MBytes. With high resolution pictures and figures, it’s easy to go beyond the limit.

Manually resizing images and deleting images that aren’t actually in the final version is time consuming and cumbersome, so arXiv LaTeX cleaner does that automatically, too. If there’s a very intricate figure you’d like to keep in high resolution, you can specify a list of images and their expected resolution.

We hope that, by making open sourcing research papers faster and safer, arXiv LaTeX cleaner will help even more researchers embrace open access and make their work freely available.

arXiv LaTeX cleaner itself is open source, so you can adapt it to your needs. If you think your adaptation would be useful for others, we’d love your contributions, too.

By Jordi Pont-Tuset, Machine Perception team

Open sourcing ClusterFuzz

Fuzzing is an automated method for detecting bugs in software that works by feeding unexpected inputs to a target program. It is effective at finding memory corruption bugs, which often have serious security implications. Manually finding these issues is both difficult and time consuming, and bugs often slip through despite rigorous code review practices. For software projects written in an unsafe language such as C or C++, fuzzing is a crucial part of ensuring their security and stability.

In order for fuzzing to be truly effective, it must be continuous, done at scale, and integrated into the development process of a software project. To provide these features for Chrome, we wrote ClusterFuzz, a fuzzing infrastructure running on over 25,000 cores. Two years ago, we began offering ClusterFuzz as a free service to open source projects through OSS-Fuzz.

Today, we’re announcing that ClusterFuzz is now open source and available for anyone to use.



We developed ClusterFuzz over eight years to fit seamlessly into developer workflows, and to make it dead simple to find bugs and get them fixed. ClusterFuzz provides end-to-end automation, from bug detection, to triage (accurate deduplication, bisection), to bug reporting, and finally to automatic closure of bug reports.

ClusterFuzz has found more than 16,000 bugs in Chrome and more than 11,000 bugs in over 160 open source projects integrated with OSS-Fuzz. It is an integral part of the development process of Chrome and many other open source projects. ClusterFuzz is often able to detect bugs hours after they are introduced and verify the fix within a day.

Check out our GitHub repository. You can try ClusterFuzz locally by following these instructions. In production, ClusterFuzz depends on some key Google Cloud Platform services, but you can use your own compute cluster. We welcome your contributions and look forward to any suggestions to help improve and extend this infrastructure. Through open sourcing ClusterFuzz, we hope to encourage all software developers to integrate fuzzing into their workflows.

By Abhishek Arya, Oliver Chang, Max Moroz, Martin Barbella and Jonathan Metzman, ClusterFuzz team

Dopamine 2.0: providing more flexibility in reinforcement learning research

Reinforcement learning (RL) has become one of the most popular fields of machine learning, and has seen a number of great advances over the last few years. As a result, there is a growing need from both researchers and educators to have access to a clear and reliable framework for RL research and education.

Last August, we announced Dopamine, our framework for flexible reinforcement learning.  For the initial version we decided to focus on a specific type of RL research: value-based agents evaluated on the Atari 2600 framework supported by the Arcade Learning Environment. We were thrilled to see how well it was received by the community, including a live coding session, its inclusion in a recently-announced benchmark for RL, considered as the top “Cool new open source project of 2018” by the Octoverse, and over 7K GitHub stars on our repository.

One of the most common requests we have received is support for more environments. This confirms what we have seen internally, where simpler environments, such as those supported by OpenAI’s Gym, are incredibly useful when testing out new algorithms. We are happy to announce Dopamine 2.0, which includes support for discrete-domain gym environments (e.g. discrete states and actions). The core of the framework remains unchanged, we have simply generalized the interface with the environment. For backwards compatibility, users will still be able to download version 1.0.

We include default configurations for two classic control environments: CartPole and Acrobot; on these environments one can train a Dopamine agent in minutes. When compared with the training time for a standard Atari 2600 game (around 5 days on a standard GPU), these environments allow researchers to iterate much faster on research ideas before testing them out on larger Atari games. We also include a Colaboratory that illustrates how to train an agent on Cartpole and Acrobot. Finally, our GymPreprocessing class serves as an example for how to use Dopamine with other custom environments.

We are excited by the new opportunities enabled by Dopamine 2.0, and look forward to seeing what the research community creates with it!

By Pablo Samuel Castro and Marc G. Bellemare, Dopamine Team

Seeking open source projects for Google Summer of Code 2019

Do you lead or represent a free or open source software organization? Are you seeking new contributors? (Who isn’t?) Do you enjoy the challenge and reward of mentoring new developers? Apply to be a mentor organization for Google Summer of Code 2019!

We are searching for open source projects and organizations to participate in the 15th annual Google Summer of Code (GSoC). GSoC is a global program that draws university student developers from around the world to contribute to open source. Each student spends three months working on a coding project, with the support of volunteer mentors, for participating open source organizations from late May to August.

Last year 1,264 students worked with 206 open source organizations. Organizations include individual smaller and medium sized open source projects as well as a number of umbrella organizations with many sub-projects under them (Python Software Foundation, CERN, Apache Software Foundation).

You can apply to be a mentoring organization for GSoC starting today. The deadline to apply is February 6 at 20:00 UTC. Organizations chosen for GSoC 2019 will be publicly announced on February 26.

Please visit the program site for more information on how to apply and review the detailed timeline of important deadlines. We also encourage you to check out the Mentor Guide and our short video on why open source projects choose to apply to be a part of the program.

Best of luck to all of the project applicants!

By Stephanie Taylor, Google Open Source

A new chapter for OSS-Fuzz

Cross-posted on the Google Security Blog.

Open source software (OSS) is extremely important to Google, and we rely on OSS in a variety of customer-facing and internal projects. We also understand the difficulty and importance of securing the open source ecosystem, and are continuously looking for ways to simplify it.

For the OSS community, we currently provide OSS-Fuzz, a free continuous fuzzing infrastructure hosted on the Google Cloud Platform. OSS-Fuzz uncovers security vulnerabilities and stability issues, and reports them directly to developers. Since launching in December 2016, OSS-Fuzz has reported over 9,000 bugs directly to open source developers.

In addition to OSS-Fuzz, Google's security team maintains several internal tools for identifying bugs in both Google internal and open source code. Until recently, these issues were manually reported to various public bug trackers by our security team and then monitored until they were resolved. Unresolved bugs were eligible for the Patch Rewards Program. While this reporting process had some success, it was overly complex. Now, by unifying and automating our fuzzing tools, we have been able to consolidate our processes into a single workflow, based on OSS-Fuzz. Projects integrated with OSS-Fuzz will benefit from being reviewed by both our internal and external fuzzing tools, thereby increasing code coverage and discovering bugs faster.

We are committed to helping open source projects benefit from integrating with our OSS-Fuzz fuzzing infrastructure. In the coming weeks, we will reach out via email to critical projects that we believe would be a good fit and support the community at large. Projects that integrate are eligible for rewards ranging from $1,000 (initial integration) up to $20,000 (ideal integration); more details are available here. These rewards are intended to help offset the cost and effort required to properly configure fuzzing for OSS projects. If you would like to integrate your project with OSS-Fuzz, please submit your project for review. Our goal is to admit as many OSS projects as possible and ensure that they are continuously fuzzed.

Once contacted, we might provide a sample fuzz target to you for easy integration. Many of these fuzz targets are generated with new technology that understands how library APIs are used appropriately. Watch this space for more details on how Google plans to further automate fuzz target creation, so that even more open source projects can benefit from continuous fuzzing.

Thank you for your continued contributions to the open source community. Let’s work together on a more secure and stable future for open source software.

By Matt Ruhstaller, TPM and Oliver Chang, Software Engineer, Google Security Team

The big reveal: Google Code-in 2018 winners and finalists

Our 9th consecutive year of Google Code-in (GCI) 2018 ended in mid-December. It was a very, very busy seven weeks for everyone – we had 3,124 students from 77 countries completing 15,323 tasks with a record 27 open source organizations!

Today, we are pleased to announce the Google Code-in 2018 Grand Prize Winners and Finalists with each organization. The 54 Grand Prize Winners from 19 countries completed an impressive 1,668 tasks between them while also helping other students during the contest.

Each of the Grand Prize Winners are invited to a four day trip to Google’s main campus and San Francisco offices in Northern California where they’ll meet Google engineers, meet one of the mentors they worked with during the contest, and enjoy some fun in California with the other winners. We look forward to seeing everyone later this year!
Country # of Winners Country # of Winners
Cameroon 1 Romania 1
Canada 1 Russian Federation 1
Czech Republic 1 Singapore 1
Georgia 1 South Africa 1
India 18 Spain 2
Indonesia 1 Sri Lanka 1
Macedonia 1 Ukraine 2
Netherlands 1 United Kingdom 6
Philippines 1 United States 9
Poland 4

Finalists

And a big congratulations to our 108 Finalists from 26 countries who completed over 2,350 tasks during the contest. The Finalists will all receive a special hoodie to commemorate their achievements in the contest. This year we had 1 student named as a finalist with 2 different organizations!

A breakdown of the countries represented by our finalists can be found below. 
Country # of Finalists Country # of Finalists
Canada 6 Philippines 1
China 2 Poland 15
Czech Republic 1 Russian Federation 2
Germany 1 Serbia 1
India 48 Singapore 2
Indonesia 2 South Korea 1
Israel 1 Spain 1
Kazakhstan 1 Sri Lanka 2
Luxembourg 1 Taiwan 1
Mauritius 2 Thailand 1
Mexico 1 United Kingdom 3
Nepal 1 United States 8
Pakistan 2 Uruguay 1

Mentors

This year we had 790 mentors dedicate their time and invaluable expertise to helping thousands of teenage students learn about open source by welcoming them into their communities. These mentors are the heart of GCI and the reason the contest continues to thrive. Mentors spend hundreds of hours answering questions, reviewing submitted tasks, and teaching students the basics and, in many cases, more advanced aspects of contributing to open source. GCI would not be possible without their enthusiasm and commitment.

We will post more statistics and fun stories that came from GCI 2018 here on the Google Open Source Blog over the next few months, so please stay tuned.

Congratulations to our Grand Prize Winners, Finalists, and all of the students who spent the last couple of months learning about, and contributing to, open source. We hope they will continue their journey in open source!

By Stephanie Taylor, Google Open Source

Wrapping up Google Code-in 2018

We are excited to announce the conclusion of the 9th annual Google Code-in (GCI), our global online contest introducing teenagers to the world of open source development. Over the years the contest has not only grown bigger, but also helped find and support talented young people around the world.

Here are some initial statistics about this year’s program:
  • Total number of students completing tasks: 3,123*
  • Total number of countries represented by students: 77
  • Percentage of girls among students: 17.9% 
Below you can see the total number of tasks completed by students year over year:
*These numbers will increase as mentors finish reviewing the final work submitted by students this morning.
Mentors from each of the 27 open source organizations are now busy reviewing the last  work submitted by participants. We look forward to sharing more statistics about the program, including countries and schools with the most student participants, in an upcoming blog post.

The mentors for each organization will spend the next couple of weeks selecting four Finalists (who will receive a hoodie too!) and their two Grand Prize Winners. Grand Prize Winners will be flown to Northern California to visit Google’s headquarters, enjoy a day of adventure in San Francisco, meet their mentors and hear talks from Google engineers.

Hearty congratulations to all the student participants for challenging themselves and making contributions to open source in the process!

Further, we’d like to thank the mentors and the organization administrators for GCI 2018. They are the heart of this program, volunteering countless hours creating tasks, reviewing student work, and helping bring students into the world of open source. Mentors teach young students about the many facets of open source development, from community standards and communicating across time zones to version control and testing. We couldn’t run this program without you! Thank you!

Stay tuned, we’ll be announcing the Grand Prize Winners and Finalists on January 7, 2019!

By Saranya Sampat, Google Open Source