Category Archives: Open Source Blog

News about Google’s open source projects and programs

The latest round of Google Open Source Peer Bonus winners

Google relies on open source software throughout our systems, much of it written by non-Googlers. We’re always looking for ways to say “thank you!” so 5 years ago we started asking Googlers to nominate open source contributors outside of the company who have made significant contributions to codebases we use or think are important. We’ve recognized more than 500 developers from 30+ countries who have contributed their time and talent to over 400 open source projects since the program’s inception in 2011.

Today we are pleased to announce the latest round of awardees, 52 individuals we’d like to recognize for their dedication to open source communities. The following is a list of everyone who gave us permission to thank them publicly:

Name Project Name Project
Philipp Hancke Adapter.js Fernando Perez Jupyter & IPython
Geoff Greer Ag Michelle Noorali Kubernetes & Helm
Dzmitry Shylovich Angular Prosper Otemuyiwa Laravel Hackathon Starter
David Kalnischkies Apt Keith Busch Linux kernel
Peter Mounce Bazel Thomas Caswell matplotlib
Yuki Yugui Sonoda Bazel Tatsuhiro Tsujikawa nghttp2
Eric Fiselier benchmark Anna Henningsen Node.js
Rob Stradling Certificate Transparency Charles Harris NumPy
Ke He Chromium Jeff Reback pandas
Daniel Micay CopperheadOS Ludovic Rousseau PCSC-Lite, CCID
Nico Huber coreboot Matti Picus PyPy
Kyösti Mälkki coreboot Salvatore Sanfilippo Redis
Jana Moudrá Dart Ralf Gommers SciPy
John Wiegley Emacs Kevin O'Connor SeaBIOS
Alex Saveau FirebaseUI-Android Sam Aaron Sonic Pi
Toke Hoiland-Jorgensen Flent Michael Tyson The Amazing Audio Engine
Hanno Böck Fuzzing Project Rob Landley Toybox
Luca Milanesio Gerrit Bin Meng U-Boot
Daniel Theophanes Go programming language Ben Noordhuis V8
Josh Snyder Go programming language Fatih Arslan vim-go
Brendan Tracey Go programming language Adam Treat WebKit
Elias Naur Go on Mobile Chris Dumez WebKit
Anthonios Partheniou Google Cloud Datalab Sean Larkin Webpack
Marcus Meissner gPhoto2 Tobias Koppers Webpack
Matt Butcher Helm Alexis La Goutte Wireshark dissector for QUIC

Congratulations to all of the awardees, past and present! Thank you for your contributions.

By Helen Hu, Open Source Programs Office

Dispatches from the latest Mercurial sprints

On March 10th-12th, the Mercurial project held one of its twice-a-year sprints in the Google Mountain View office. Mercurial is a distributed version control system, used by Google, W3C, OpenJDK and Mozilla among others. We had 40 developers in attendance, some from companies with large Mercurial deployments and some individual contributors who volunteer in their spare time.

One of the major themes we discussed was user-friendliness. Mercurial developers work hard to keep the command-line interface backwards compatible, but at the same time, we would like to make progress by smoothing out some rough edges. We discussed how we can provide a better user interface for users to opt-in to without breaking the backwards compatibility constraint. We also talked about how to make Mercurial’s Changeset Evolution feature easier to use.

We considered moving Mercurial past SHA1 for revision identification, to enhance security and integrity of Mercurial repositories in light of recent SHA1 exploits. A rough consensus on a plan started to emerge, and design docs should start to circulate in the next month or so.

We also talked about performance, such as new storage layers that would scale more effectively and work better with clones that only contain a partial repository history, a key requirement for Mercurial adoption in enterprise environments with large repositories, like Google.

If you are interested in finding out more about Mercurial (or perhaps you’d like to contribute!) you can find our mailing list information here.

By Martin von Zweigbergk and Augie Fackler, Software Engineers

An Upgrade to SyntaxNet, New Models and a Parsing Competition

At Google, we continuously improve the language understanding capabilities used in applications ranging from generation of email responses to translation. Last summer, we open-sourced SyntaxNet, a neural-network framework for analyzing and understanding the grammatical structure of sentences. Included in our release was Parsey McParseface, a state-of-the-art model that we had trained for analyzing English, followed quickly by a collection of pre-trained models for 40 additional languages, which we dubbed Parsey's Cousins. While we were excited to share our research and to provide these resources to the broader community, building machine learning systems that work well for languages other than English remains an ongoing challenge. We are excited to announce a few new research resources, available now, that address this problem.

SyntaxNet Upgrade
We are releasing a major upgrade to SyntaxNet. This upgrade incorporates nearly a year’s worth of our research on multilingual language understanding, and is available to anyone interested in building systems for processing and understanding text. At the core of the upgrade is a new technology that enables learning of richly layered representations of input sentences. More specifically, the upgrade extends TensorFlow to allow joint modeling of multiple levels of linguistic structure, and to allow neural-network architectures to be created dynamically during processing of a sentence or document.

Our upgrade makes it, for example, easy to build character-based models that learn to compose individual characters into words (e.g. ‘c-a-t’ spells ‘cat’). By doing so, the models can learn that words can be related to each other because they share common parts (e.g. ‘cats’ is the plural of ‘cat’ and shares the same stem; ‘wildcat’ is a type of ‘cat’). Parsey and Parsey’s Cousins, on the other hand, operated over sequences of words. As a result, they were forced to memorize words seen during training and relied mostly on the context to determine the grammatical function of previously unseen words.

As an example, consider the following (meaningless but grammatically correct) sentence:
This sentence was originally coined by Andrew Ingraham who explained: “You do not know what this means; nor do I. But if we assume that it is English, we know that the doshes are distimmed by the gostak. We know too that one distimmer of doshes is a gostak." Systematic patterns in morphology and syntax allow us to guess the grammatical function of words even when they are completely novel: we understand that ‘doshes’ is the plural of the noun ‘dosh’ (similar to the ‘cats’ example above) or that ‘distim’ is the third person singular of the verb distim. Based on this analysis we can then derive the overall structure of this sentence even though we have never seen the words before.

ParseySaurus
To showcase the new capabilities provided by our upgrade to SyntaxNet, we are releasing a set of new pretrained models called ParseySaurus. These models use the character-based input representation mentioned above and are thus much better at predicting the meaning of new words based both on their spelling and how they are used in context. The ParseySaurus models are far more accurate than Parsey’s Cousins (reducing errors by as much as 25%), particularly for morphologically-rich languages like Russian, or agglutinative languages like Turkish and Hungarian. In those languages there can be dozens of forms for each word and many of these forms might never be observed during training - even in a very large corpus.

Consider the following fictitious Russian sentence, where again the stems are meaningless, but the suffixes define an unambiguous interpretation of the sentence structure:
Even though our Russian ParseySaurus model has never seen these words, it can correctly analyze the sentence by inspecting the character sequences which constitute each word. In doing so, the system can determine many properties of the words (notice how many more morphological features there are here than in the English example). To see the sentence as ParseySaurus does, here is a visualization of how the model analyzes this sentence:
Each square represents one node in the neural network graph, and lines show the connections between them. The left-side “tail” of the graph shows the model consuming the input as one long string of characters. These are intermittently passed to the right side, where the rich web of connections shows the model composing words into phrases and producing a syntactic parse. Check out the full-size rendering here.

A Competition
You might be wondering whether character-based modeling are all we need or whether there are other techniques that might be important. SyntaxNet has lots more to offer, like beam search and different training objectives, but there are of course also many other possibilities. To find out what works well in practice we are helping co-organize, together with Charles University and other colleagues, a multilingual parsing competition at this year’s Conference on Computational Natural Language Learning (CoNLL) with the goal of building syntactic parsing systems that work well in real-world settings and for 45 different languages.

The competition is made possible by the Universal Dependencies (UD) initiative, whose goal is to develop cross-linguistically consistent treebanks. Because machine learned models can only be as good as the data that they have access to, we have been contributing data to UD since 2013. For the competition, we partnered with UD and DFKI to build a new multilingual evaluation set consisting of 1000 sentences that have been translated into 20+ different languages and annotated by linguists with parse trees. This evaluation set is the first of its kind (in the past, each language had its own independent evaluation set) and will enable more consistent cross-lingual comparisons. Because the sentences have the same meaning and have been annotated according to the same guidelines, we will be able to get closer to answering the question of which languages might be harder to parse.

We hope that the upgraded SyntaxNet framework and our the pre-trained ParseySaurus models will inspire researchers to participate in the competition. We have additionally created a tutorial showing how to load a Docker image and train models on the Google Cloud Platform, to facilitate participation by smaller teams with limited resources. So, if you have an idea for making your own models with the SyntaxNet framework, sign up to compete! We believe that the configurations that we are releasing are a good place to start, but we look forward to seeing how participants will be able to extend and improve these models or perhaps create better ones!

Thanks to everyone involved who made this competition happen, including our collaborators at UD-Pipe, who provide another baseline implementation to make it easy to enter the competition. Happy parsing from the main developers, Chris Alberti, Daniel Andor, Ivan Bogatyy, Mark Omernick, Zora Tung and Ji Ma!

By David Weiss and Slav Petrov, Research Scientists

Google Summer of Code 2017 student applications are open!

Are you a university student looking to learn more about open source software development? Consider applying to Google Summer of Code (GSoC) for a chance to spend your break coding on an open source project.

vertical GSoC logo.jpg


For the 13th straight year GSoC will give students from around the world the opportunity to learn the ins and outs of open source software development while working from their home. Students will receive a stipend for their successful contributions to allow them to focus on their coding during the program.

Mentors are paired with the students to help address technical questions and to monitor their progress throughout the program. Former GSoC participants have told us that the real-world experience they’ve gained during the program has not only sharpened their technical skills, but has also boosted their confidence, broadened their professional network and enhanced their resumes.

Interested students can submit proposals on the program site now through Monday, April 3 at 16:00 UTC. The first step is to search through the 201 open source organizations and review the “Project ideas” for the organizations that appeal to you. Next, reach out to the organizations to introduce yourself and determine if your skills and interests are a good match with their organization.




Since spots are limited, we recommend writing a strong project proposal and submitting a draft early to receive feedback from the organization which will help increase your chances of selection. Our Student Manual, written by former students and mentors, provides excellent helpful advice to get you started with choosing an organization and crafting a great proposal.

For information throughout the application period and beyond, visit the Google Open Source Blog, join our Google Summer of Code discussion lists or join us on Internet Relay Chat (IRC) at #gsoc on Freenode. Be sure to read the Program Rules, Timeline and FAQ, all available on the program site, for more information about Google Summer of Code.

Good luck to all the open source coders who apply, and remember to submit your proposals early — you only have until Monday, April 3 at 16:00 UTC!

By Stephanie Taylor, Google Summer of Code Program Manager

Announcing Google Radio PHY Test, aka “Graphyte”, as part of the Chromium Project

With many different Chromebook models for sale from several different OEMs, the Chrome OS Factory team interfaces with many different Contract Manufacturers (CMs), Original Device Manufacturers (ODMs), and factory teams. The Google Chromium OS Factory Software Platform, a suite of factory tools provided to Chrome OS partners, allows any factory team to quickly bring up a Chrome OS manufacturing test line.

Today, we are announcing that this platform has been extended to remove the friction of bringing up wireless verification test systems with a component called Google Radio PHY Test or “Graphyte.” Graphyte is an open source software framework that can be used and extended by the wireless ecosystem of chipset companies, test solution providers, and wireless device manufacturers, as opposed to the traditional approach of vendor-specific solutions. It is developed in Python and capable of running on Linux and Chrome OS test stations with an initial focus on Wi-Fi, Bluetooth, and 802.15.4 physical layer verification.

Verifying that a wireless device is working properly requires chipset- and instrument-specific software which coordinate transmitting and measuring power and signal quality across channels, bandwidths, and data rates. Graphyte provides high-level API abstractions for controlling wireless chipsets and test instruments, allowing anyone to develop a “plugin” for a given chipset or instrument.

Graphyte architecture.

We’ve worked closely with industry leaders like Intel and LitePoint to ensure the Graphyte APIs have the right level of abstraction, and it is already being used in production on multiple manufacturing lines and several different products.

To get started, use Git to clone our repository. You can learn more by reading the Graphyte User Manual and checking out the example of how to use Graphyte in a real test. You can get involved by joining our mailing list. If you’d like to contribute, please follow the Chromium OS Developer Guide.

To get started with the LitePoint Graphyte plugin, please contact LitePoint directly. To get started with the Intel Graphyte plugin, please contact Intel directly.

Happy testing!

By Kurt Williams, Technical Program Manager

Announcing Guetzli: A New Open Source JPEG Encoder

Crossposted on the Google Research Blog

At Google, we care about giving users the best possible online experience, both through our own services and products and by contributing new tools and industry standards for use by the online community. That’s why we’re excited to announce Guetzli, a new open source algorithm that creates high quality JPEG images with file sizes 35% smaller than currently available methods, enabling webmasters to create webpages that can load faster and use even less data.

Guetzli [guɛtsli] — cookie in Swiss German — is a JPEG encoder for digital images and web graphics that can enable faster online experiences by producing smaller JPEG files while still maintaining compatibility with existing browsers, image processing applications and the JPEG standard. From the practical viewpoint this is very similar to our Zopfli algorithm, which produces smaller PNG and gzip files without needing to introduce a new format; and different than the techniques used in RNN-based image compression, RAISR, and WebP, which all need client and ecosystem changes for compression gains at internet scale.

The visual quality of JPEG images is directly correlated to its multi-stage compression process: color space transform, discrete cosine transform, and quantization. Guetzli specifically targets the quantization stage in which the more visual quality loss is introduced, the smaller the resulting file. Guetzli strikes a balance between minimal loss and file size by employing a search algorithm that tries to overcome the difference between the psychovisual modeling of JPEG's format, and Guetzli’s psychovisual model, which approximates color perception and visual masking in a more thorough and detailed way than what is achievable by simpler color transforms and the discrete cosine transform. However, while Guetzli creates smaller image file sizes, the tradeoff is that these search algorithms take significantly longer to create compressed images than currently available methods.

orig-libjpeg-guetzli.png
Figure 1. 16x16 pixel synthetic example of  a phone line  hanging against a blue sky — traditionally a case where JPEG compression algorithms suffer from artifacts. Uncompressed original is on the left. Guetzli (on the right) shows less ringing artefacts than libjpeg (middle) and has a smaller file size.
And while Guetzli produces smaller image file sizes without sacrificing quality, we additionally found that in experiments where compressed image file sizes are kept constant that human raters consistently preferred the images Guetzli produced over libjpeg images, even when the libjpeg files were the same size or even slightly larger. We think this makes the slower compression a worthy tradeoff.

montage-cats-zoom-eye2.png
Figure 2. 20x24 pixel zoomed areas from a picture of a cat’s eye. Uncompressed original on the left. Guetzli (on the right) shows less ringing artefacts than libjpeg (middle) without requiring a larger file size.
It is our hope that webmasters and graphic designers will find Guetzli useful and apply it to their photographic content, making users’ experience smoother on image-heavy websites in addition to reducing load times and bandwidth costs for mobile users. Last, we hope that the new explicitly psychovisual approach in Guetzli will inspire further image and video compression research.

By Robert Obryk and Jyrki Alakuijala, Software Engineers, Google Research Europe

Getting ready for Google Summer of Code 2017

Spring is just around the corner here in the Northern Hemisphere and Google Summer of Code is fast approaching. If you are a student interested in participating this year, now is the time to prepare -- read on for tips on how to get ready.

This year we’ve accepted 201 open source organizations into the program, nearly 40 of which are new to the program. The organizations cover a wide range of topics including (but certainly not limited to!):

  • Operating systems
  • Web application frameworks
  • Healthcare and bioinformatics
  • Music and graphic design
  • Machine learning
  • Robotics
  • Security




How should you prepare for Google Summer of Code?

While student applications don’t open until March 20th at 16:00 UTC, you need to decide which projects you’re interested in and what you’ll propose. You should also communicate with those projects to learn more before you apply.

Start by looking at the list of participating projects and organizations. You can explore by searching for specific names or technologies, or filtering by topics you are interested in. Follow the “Learn More” link through to each organization’s page for additional information.

Once you’ve identified the organizations that you’re interested in, take a look at their ideas list to get a sense of the specific projects you could work on. Typically, you will choose a project from that list and write a proposal based on that idea, but you could also propose something that’s not on that list.

You should reach out to the organizations after you’ve decided what you want to work on. Doing this can make the difference between a good application and a great application.

Whatever you do, don’t wait until March 20th to begin preparing for Google Summer of Code! History has shown that students who reach out to organizations before the start of the application period have a higher chance of being accepted into the program, as they have had more time to talk to the organizations and understand what they are looking for with the project.

If you have any questions along the way, take a look at the Student Manual, FAQ and Timeline. If you can’t find the answer to your question, try taking your question to the mailing list.

By Josh Simmons, Open Source Programs Office

Another option for file sharing

Originally posted on the Google Security Blog

Existing mechanisms for file sharing are so fragmented that people waste time on multi-step copying and repackaging. With the new open source project Upspin, we aim to improve the situation by providing a global name space to name all your files. Given an Upspin name, a file can be shared securely, copied efficiently without "download" and "upload", and accessed by anyone with permission from anywhere with a network connection.

Our target audience is personal users, families, or groups of friends. Although Upspin might have application in enterprise environments, we think that focusing on the consumer case enables easy-to-understand and easy-to-use sharing.

File names begin with the user's email address followed by a slash-separated Unix-like path name:

ann@example.com/dir/file.

Any user with appropriate permission can access the contents of this file by using Upspin services to evaluate the full path name, typically via a FUSE filesystem so that unmodified applications just work. Upspin names usually identify regular static files and directories, but may point to dynamic content generated by devices such as sensors or services.

If the user wishes to share a directory (the unit at which sharing privileges are granted), she adds a file called Access to that directory. In that file she describes the rights she wishes to grant and the users she wishes to grant them to. For instance,

read: joe@here.com, mae@there.com

allows Joe and Mae to read any of the files in the directory holding the Access file, and also in its subdirectories. As well as limiting who can fetch bytes from the server, this access is enforced end-to-end cryptographically, so cleartext only resides on Upspin clients, and use of cloud storage does not extend the trust boundary.

Upspin looks a bit like a global file system, but its real contribution is a set of interfaces, protocols, and components from which an information management system can be built, with properties such as security and access control suited to a modern, networked world. Upspin is not an "app" or a web service, but rather a suite of software components, intended to run in the network and on devices connected to it, that together provide a secure, modern information storage and sharing network. Upspin is a layer of infrastructure that other software and services can build on to facilitate secure access and sharing. This is an open source contribution, not a Google product. We have not yet integrated with the Key Transparency server, though we expect to eventually, and for now use a similar technique of securely publishing all key updates. File storage is inherently an archival medium without forward secrecy; loss of the user's encryption keys implies loss of content, though we do provide for key rotation.

It’s early days, but we’re encouraged by the progress and look forward to feedback and contributions. To learn more, see the GitHub repository at Upspin.

By Andrew Gerrand, Eric Grosse, Rob Pike, Eduardo Pinheiro and Dave Presotto, Google Software Engineers

By maintainers, for maintainers: Wontfix_Cabal

The Google Open Source Programs Office likes to highlight events we support, organize, or speak at. In this case, Google’s own Jess Frazelle was responsible for running a unique event for open source maintainers.

This year I helped organize the first inaugural Wontfix_Cabal. The conference was organized by open source software maintainers for open source software maintainers. Our initial concept was an unconference where attendees could discuss topics candidly with their peers from other open source communities.

The idea for the event stemmed from the response to a blog post I published about closing pull requests. The response was overwhelming, with many maintainers commiserating and sharing lessons they had learned. It seemed like we could all learn a lot from our peers in other projects -- if we had the space to do so -- and it was clear that people needed a place to vent.

Major thanks to Katrina Owen and Brandon Keepers from GitHub who jumped right in and provided the venue we needed to make this happen. Without their support this would’ve never become a reality!

It was an excellent first event and the topics discussed were wide ranging, including:
  • How to deal with unmaintained projects
  • Collecting metrics to judge project health
  • Helping newcomers
  • Dealing with backlogs
  • Coping with, and minimizing, toxic behavior in our communities


The discussion around helping newcomers focused on creating communities with welcoming and productive cultures right from the start. I was fascinated to learn that some projects pre-fill issues before going public so as to set the tone for the future of the project. Another good practice is clearly defining how one becomes a maintainer or gets commit access. There should be clear rules in place so people know what they have to do to succeed.

Another discussion I really liked focused on “saying no.” Close fast and close early was a key takeaway. There’s no sense in letting a contribution sit waiting when you know it will never be accepted. Multiple projects found that having a bot give the hard news was always better than having the maintainer do it. This way it is not personal, just a regular part of the process.

One theme seen in multiple sessions: “Being kind is not the same as being nice.” The distinction here is that being nice comes from a place of fear and leads people to bend over backwards just to please. Being kind comes from a place of strength, from doing the right thing.

Summaries of many of the discussions have been added to the GitHub repo if you would like to read more.

After the event concluded many maintainers got right to work, putting what they had learned into practice. For instance, Rust got help from the Google open source fuzzing team.


Our goal was to put together a community of maintainers that could support and learn from each other. When I saw Linux kernel maintainers talking to people who work on Node and JavaScript, I knew we had achieved that goal. Laura Abbott, one of those kernel developers, wrote a blog post about the experience.

Not only was the event useful, it was also a lot of fun. Meeting maintainers, people who care a great deal about open source software, from such a diverse group of projects was great. Overall, I think our initial run was a success! Follow us on Twitter to find out about future events.

By Jess Frazelle, Software Engineer

Introducing Python Fire, a library for automatically generating command line interfaces

Today we are pleased to announce the open-sourcing of Python Fire. Python Fire generates command line interfaces (CLIs) from any Python code. Simply call the Fire function in any Python program to automatically turn that program into a CLI. The library is available from pypi via `pip install fire`, and the source is available on GitHub.

Python Fire will automatically turn your code into a CLI without you needing to do any additional work. You don't have to define arguments, set up help information, or write a main function that defines how your code is run. Instead, you simply call the `Fire` function from your main module, and Python Fire takes care of the rest. It uses inspection to turn whatever Python object you give it -- whether it's a class, an object, a dictionary, a function, or even a whole module -- into a command line interface, complete with tab completion and documentation, and the CLI will stay up-to-date even as the code changes.

To illustrate this, let's look at a simple example.

#!/usr/bin/env python
import fire

class Example(object):
def hello(self, name='world'):
"""Says hello to the specified name."""
return 'Hello {name}!'.format(name=name)

def main():
fire.Fire(Example)

if __name__ == '__main__':
main()

When the Fire function is run, our command will be executed. Just by calling Fire, we can now use the Example class as if it were a command line utility.

$ ./example.py hello
Hello world!
$ ./example.py hello David
Hello David!
$ ./example.py hello --name=Google
Hello Google!

Of course, you can continue to use this module like an ordinary Python library, enabling you to use the exact same code both from Bash and Python. If you're writing a Python library, then you no longer need to update your main method or client when experimenting with it; instead you can simply run the piece of your library that you're experimenting with from the command line. Even as the library changes, the command line tool stays up to date.

At Google, engineers use Python Fire to generate command line tools from Python libraries. We have an image manipulation tool built by using Fire with the Python Imaging Library, PIL. In Google Brain, we use an experiment management tool built with Fire, allowing us to manage experiments equally well from Python or from Bash.

Every Fire CLI comes with an interactive mode. Run the CLI with the `--interactive` flag to launch an IPython REPL with the result of your command, as well as other useful variables already defined and ready to use. Be sure to check out Python Fire's documentation for more on this and the other useful features Fire provides.

Between Python Fire's simplicity, generality, and power, we hope you find it a useful library for your own projects.

By David Bieber, Software Engineer on Google Brain