Tag Archives: Open source

Introducing a Web Component and Data API for Quick, Draw!


Over the past couple years, the Creative Lab in collaboration with the Handwriting Recognition team have released a few experiments in the realm of “doodle” recognition.  First, in 2016, there was Quick, Draw!, which uses a neural network to guess what you’re drawing.  Since Quick, Draw! launched we have collected over 1 billion drawings across 345 categories.  In the wake of that popularity, we open sourced a collection of 50 million drawings giving developers around the world access to the data set and the ability to conduct research with it.

"The different ways in which people draw are like different notes in some universally human scale" - Ian Johnson, UX Engineer @ Google

Since the initial dataset was released, it has been incredible to see how graphs, t-sne clusters, and simply overlapping millions of these doodles have given us the ability to infer interesting human behaviors, across different cultures.  One example, from the Quartz study, is that 86% of Americans (from a sample of 50,000) draw their circles counterclockwise, while 80% of Japanese (from a sample of 800) draw them clockwise. Part of this pattern in behavior can be attributed to the strict stroke order in Japanese writing, from the top left to the bottom right.


It’s also interesting to see how the data looks when it’s overlaid by country, as Kyle McDonald did, when he discovered that some countries draw their chairs in perspective while others draw them straight on.


On the more fun, artistic spectrum, there are some simple but clever uses of the data like Neil Mendoza’s face tracking experiment and Deborah Schmidt’s letter collages.
See the video here of Neil Mendoza mapping Quick, Draw! facial features to your own face in front of a webcam


See the process video here of Deborah Schmidt packing QuickDraw data into letters using OpenFrameworks
Some handy tools have also been released from the community since the release of all this data, and one of those that we’re releasing now is a Polymer component that allows you to display a doodle in your web-based project with one line of markup:

The Polymer component is coupled with a Data API that layers a massive file directory (50 million files) and returns a JSON object or an HTML canvas rendering for each drawing.  Without downloading all the data, you can start creating right away in prototyping your ideas.  We’ve also provided instructions for how to host the data and API yourself on Google Cloud Platform (for more serious projects that demand a higher request limit).  

One really handy tool when hosting an API on Google Cloud is Cloud Endpoints.  It allowed us to launch a demo API with a quota limit and authentication via an API key.  

By defining an OpenAPI specification (here is the Quick, Draw! Data API spec) and adding these three lines to our app.yaml file, an Extensible Service Proxy (ESP) gets deployed with our API backend code (more instructions here):
endpoints_api_service:
name: quickdrawfiles.appspot.com
rollout_strategy: managed
Based on the OpenAPI spec, documentation is also automatically generated for you:


We used a public Google Group as an access control list, so anyone who joins can then have the API available in their API library.
The Google Group used as an Access Control List
This component and Data API will make it easier for  creatives out there to manipulate the data for their own research.  Looking to the future, a potential next step for the project could be to store everything in a single database for more complex queries (i.e. “give me an recognized drawing from China in March 2017”).  Feedback is always welcome, and we hope this inspires even more types of projects using the data! More details on the project and the incredible research projects done using it can be found on our GitHub repo

By Nick Jonas, Creative Technologist, Creative Lab

Editor's Note: Some may notice that this isn’t the only dataset we’ve open sourced recently! You can find many more datasets in our open source project directory.

Google Summer of Code: 15 years strong!

Google Open Source is proud to announce Google Summer of Code (GSoC) 2019 – the 15th year of the program! We look forward to introducing the 15th batch of student developers to the world of open source and matching them with open source projects.

Over the last 14 years GSoC has provided over 14,000 university students from 109 countries with an opportunity to hone their skills by contributing to open source projects during their summer break. Participants gain invaluable experience working directly with mentors on open source projects, and earn a stipend upon successful completion of their project.

We’re excited to keep the tradition going! Applications for interested open source project organizations open on January 15, 2019, and student applications open March 25.

Are you an open source project interested in learning more? Visit the program site and read the mentor guide to learn more about what it means to be a mentor organization and how to prepare your community and your application. We welcome all types of organizations – large and small – and are eager to involve first time projects. Each year, about 20% of the organizations we accept are completely new to GSoC.

Are you a university student keen on learning about how to prepare for the 2019 GSoC program? It’s never too early to start thinking about your proposal or about what type of open source organization you may want to work with. You should read the student guide for important tips on preparing your proposal and what to consider if you wish to apply for the program in March. You can also get inspired by checking out the 200+ organizations that participated in Google Summer of Code 2018 as well as the projects that students worked on.

We encourage you to explore other resources and you can learn more on the program website.

By Stephanie Taylor, GSoC Program Lead

Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing



One of the biggest challenges in natural language processing (NLP) is the shortage of training data. Because NLP is a diversified field with many distinct tasks, most task-specific datasets contain only a few thousand or a few hundred thousand human-labeled training examples. However, modern deep learning-based NLP models see benefits from much larger amounts of data, improving when trained on millions, or billions, of annotated training examples. To help close this gap in data, researchers have developed a variety of techniques for training general purpose language representation models using the enormous amount of unannotated text on the web (known as pre-training). The pre-trained model can then be fine-tuned on small-data NLP tasks like question answering and sentiment analysis, resulting in substantial accuracy improvements compared to training on these datasets from scratch.

This week, we open sourced a new technique for NLP pre-training called Bidirectional Encoder Representations from Transformers, or BERT. With this release, anyone in the world can train their own state-of-the-art question answering system (or a variety of other models) in about 30 minutes on a single Cloud TPU, or in a few hours using a single GPU. The release includes source code built on top of TensorFlow and a number of pre-trained language representation models. In our associated paper, we demonstrate state-of-the-art results on 11 NLP tasks, including the very competitive Stanford Question Answering Dataset (SQuAD v1.1).

What Makes BERT Different?
BERT builds upon recent work in pre-training contextual representations — including Semi-supervised Sequence Learning, Generative Pre-Training, ELMo, and ULMFit. However, unlike these previous models, BERT is the first deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus (in this case, Wikipedia).

Why does this matter? Pre-trained representations can either be context-free or contextual, and contextual representations can further be unidirectional or bidirectional. Context-free models such as word2vec or GloVe generate a single word embedding representation for each word in the vocabulary. For example, the word “bank” would have the same context-free representation in “bank account” and “bank of the river.” Contextual models instead generate a representation of each word that is based on the other words in the sentence. For example, in the sentence “I accessed the bank account,” a unidirectional contextual model would represent “bank” based on “I accessed the” but not “account.” However, BERT represents “bank” using both its previous and next context — “I accessed the ... account” — starting from the very bottom of a deep neural network, making it deeply bidirectional.

A visualization of BERT’s neural network architecture compared to previous state-of-the-art contextual pre-training methods is shown below. The arrows indicate the information flow from one layer to the next. The green boxes at the top indicate the final contextualized representation of each input word:
BERT is deeply bidirectional, OpenAI GPT is unidirectional, and ELMo is shallowly bidirectional.
The Strength of Bidirectionality
If bidirectionality is so powerful, why hasn’t it been done before? To understand why, consider that unidirectional models are efficiently trained by predicting each word conditioned on the previous words in the sentence. However, it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that’s being predicted to indirectly “see itself” in a multi-layer model.

To solve this problem, we use the straightforward technique of masking out some of the words in the input and then condition each word bidirectionally to predict the masked words. For example:
While this idea has been around for a very long time, BERT is the first time it was successfully used to pre-train a deep neural network.

BERT also learns to model relationships between sentences by pre-training on a very simple task that can be generated from any text corpus: Given two sentences A and B, is B the actual next sentence that comes after A in the corpus, or just a random sentence? For example:
Training with Cloud TPUs
Everything that we’ve described so far might seem fairly straightforward, so what’s the missing piece that made it work so well? Cloud TPUs. Cloud TPUs gave us the freedom to quickly experiment, debug, and tweak our models, which was critical in allowing us to move beyond existing pre-training techniques. The Transformer model architecture, developed by researchers at Google in 2017, also gave us the foundation we needed to make BERT successful. The Transformer is implemented in our open source release, as well as the tensor2tensor library.

Results with BERT
To evaluate performance, we compared BERT to other state-of-the-art NLP systems. Importantly, BERT achieved all of its results with almost no task-specific changes to the neural network architecture. On SQuAD v1.1, BERT achieves 93.2% F1 score (a measure of accuracy), surpassing the previous state-of-the-art score of 91.6% and human-level score of 91.2%:
BERT also improves the state-of-the-art by 7.6% absolute on the very challenging GLUE benchmark, a set of 9 diverse Natural Language Understanding (NLU) tasks. The amount of human-labeled training data in these tasks ranges from 2,500 examples to 400,000 examples, and BERT substantially improves upon the state-of-the-art accuracy on all of them:
Making BERT Work for You
The models that we are releasing can be fine-tuned on a wide variety of NLP tasks in a few hours or less. The open source release also includes code to run pre-training, although we believe the majority of NLP researchers who use BERT will never need to pre-train their own models from scratch. The BERT models that we are releasing today are English-only, but we hope to release models which have been pre-trained on a variety of languages in the near future.

The open source TensorFlow implementation and pointers to pre-trained BERT models can be found at http://goo.gl/language/bert. Alternatively, you can get started using BERT through Colab with the notebook “BERT FineTuning with Cloud TPUs.”

You can also read our paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" for more details.

Source: Google AI Blog


Introducing AdaNet: Fast and Flexible AutoML with Learning Guarantees



Ensemble learning, the art of combining different machine learning (ML) model predictions, is widely used with neural networks to achieve state-of-the-art performance, benefitting from a rich history and theoretical guarantees to enable success at challenges such as the Netflix Prize and various Kaggle competitions. However, they aren’t used much in practice due to long training times, and the ML model candidate selection requires its own domain expertise. But as computational power and specialized deep learning hardware such as TPUs become more readily available, machine learning models will grow larger and ensembles will become more prominent. Now, imagine a tool that automatically searches over neural architectures, and learns to combine the best ones into a high-quality model.

Today, we’re excited to share AdaNet, a lightweight TensorFlow-based framework for automatically learning high-quality models with minimal expert intervention. AdaNet builds on our recent reinforcement learning and evolutionary-based AutoML efforts to be fast and flexible while providing learning guarantees. Importantly, AdaNet provides a general framework for not only learning a neural network architecture, but also for learning to ensemble to obtain even better models.

AdaNet is easy to use, and creates high-quality models, saving ML practitioners the time normally spent selecting optimal neural network architectures, implementing an adaptive algorithm for learning a neural architecture as an ensemble of subnetworks. AdaNet is capable of adding subnetworks of different depths and widths to create a diverse ensemble, and trade off performance improvement with the number of parameters.
AdaNet adaptively growing an ensemble of neural networks. At each iteration, it measures the ensemble loss for each candidate, and selects the best one to move onto the next iteration.
Fast and Easy to Use
AdaNet implements the TensorFlow Estimator interface, which greatly simplifies machine learning programming by encapsulating training, evaluation, prediction and export for serving. It integrates with open-source tools like TensorFlow Hub modules, TensorFlow Model Analysis, and Google Cloud’s Hyperparameter Tuner. Distributed training support significantly reduces training time, and scales linearly with available CPUs and accelerators (e.g. GPUs).
AdaNet’s accuracy (y-axis) per train step (x-axis) on CIFAR-100. The blue line is accuracy on the training set, and red line is performance on the test set. A new subnetwork begins training every million steps, and eventually improves the performance of the ensemble. The grey and green lines are the accuracies of the ensemble before adding the new subnetwork.
Because TensorBoard is one of the best TensorFlow features for visualizing model metrics during training, AdaNet integrates seamlessly with it in order to monitor subnetwork training, ensemble composition, and performance. When AdaNet is done training, it exports a SavedModel that can be deployed with TensorFlow Serving.

Learning Guarantees
Building an ensemble of neural networks has several challenges: What are the best subnetwork architectures to consider? Is it best to reuse the same architectures or encourage diversity? While complex subnetworks with more parameters will tend to perform better on the training set, they may not generalize to unseen data due to their greater complexity. These challenges stem from evaluating model performance. We could evaluate performance on a hold-out set split from the training set, but in doing so would reduce the number of examples one can use for training the neural network.

Instead, AdaNet’s approach (presented in “AdaNet: Adaptive Structural Learning of Artificial Neural Networks” at ICML 2017) is to optimize an objective that balances the trade-offs between the ensemble’s performance on the training set and its ability to generalize to unseen data. The intuition is for the ensemble to include a candidate subnetwork only when it improves the ensemble’s training loss more than it affects its ability to generalize. This guarantees that:
  1. The generalization error of the ensemble is bounded by its training error and complexity.
  2. By optimizing this objective, we are directly minimizing this bound.
A practical benefit of optimizing this objective is that it eliminates the need for a hold-out set for choosing which candidate subnetworks to add to the ensemble. This has the added benefit of enabling the use of more training data for training the subnetworks. To learn more, please walk through our tutorial about the AdaNet objective.

Extensible
We believe that the key to making a useful AutoML framework for both research and production use is to not only provide sensible defaults, but to also allow users to try their own subnetwork/model definitions. As a result, machine learning researchers, practitioners, and enthusiasts are invited to define their own AdaNet adanet.subnetwork.Builder using high level TensorFlow APIs like tf.layers.

Users who have already integrated a TensorFlow model in their system can easily convert their TensorFlow code into an AdaNet subnetwork, and use the adanet.Estimator to boost model performance while obtaining learning guarantees. AdaNet will explore their defined search space of candidate subnetworks and learn to ensemble the subnetworks. For instance, we took an open-source implementation of a NASNet-A CIFAR architecture, transformed it into a subnetwork, and improved upon CIFAR-10 state-of-the-art results after eight AdaNet iterations. Furthermore, our model achieves this result with fewer parameters:
Performance of a NASNet-A model as presented in Zoph et al., 2018 versus AdaNet learning to combine small NASNet-A subnetworks on CIFAR-10.
Users are also invited to use their own custom loss functions as part of the AdaNet objective via canned or custom tf.contrib.estimator.Heads in order to train regression, classification, and multi-task learning problems.

Users can also fully define the search space of candidate subnetworks to explore by extending the adanet.subnetwork.Generator class. This allows them to grow or reduce their search space based on their available hardware. The search space of subnetworks can be as simple as duplicating the same subnetwork configuration with different random seeds, to training dozens of subnetworks with different hyperparameter combinations, and letting AdaNet choose the one to include in the final ensemble.

If you’re interested in trying AdaNet for yourself, please check out our Github repo, and walk through the tutorial notebooks. We’ve included a few working examples using dense layers and convolutions to get you started. AdaNet is an ongoing research project, and we welcome contributions. We’re excited to see how AdaNet can help the research community.

Acknowledgements
This project was only possible thanks to the members of the core team including Corinna Cortes, Mehryar Mohri, Xavi Gonzalvo, Charles Weill, Vitaly Kuznetsov, Scott Yak, and Hanna Mazzawi. We also extend a special thanks to our collaborators, residents and interns Gus Kristiansen, Galen Chuang, Ghassen Jerfel, Vladimir Macko, Ben Adlam, Scott Yang and the many others at Google who helped us test it out.

Source: Google AI Blog


Google Code-in 2018 contest for teenagers begins today

Today marks the start of the 9th consecutive year of Google Code-in (GCI). This is the biggest and best contest ever and we hope you’ll join us for the fun!

What is Google Code-in?

Our global, online contest introducing students to open source development. The contest runs for 7 weeks until December 12, 2018.

Who can register?

Pre-university students ages 13-17 that have their parent or guardian’s permission to register for the contest.

How do students register and participate?

Students can register for the contest beginning today at g.co/gci. Once students have registered and the parental consent form has been submitted and approved by Program Administrators students can choose which contest “task” they want to work on first. Students choose the task they find interesting from a list of thousands of available tasks created by 27 participating open source organizations. Tasks take an average of 3-5 hours to complete. There are even beginner tasks that are a wonderful way for students to get started in the contest.

The task categories are:
  • Coding
  • Design
  • Documentation/Training
  • Outreach/Research
  • Quality Assurance

Why should students participate?

Students not only have the opportunity to work on a real open source software project, thus gaining invaluable skills and experience, but they also have the opportunity to be a part of the open source community. Mentors are readily available to help answer their questions while they work through the tasks.

Google Code-in is a contest so there are prizes! Complete one task and receive a digital certificate, three completed tasks and you’ll also get a fun Google t-shirt. Finalists earn the coveted hoodie. Grand Prize winners (2 from each organization) will receive a trip to Google headquarters in California!

Details

Over the last 8 years, more than 8,100 students from 107 countries have successfully completed over 40,000 tasks in GCI. Curious? Learn more about GCI by checking out the Contest Rules and FAQs. And please visit our contest site and read the Getting Started Guide.

Teachers, if you are interested in getting your students involved in Google Code-in we have resources available to help you get started.

By Stephanie Taylor, Google Open Source

Building great open source documentation

When developers use, choose, and contribute to open source software, effective documentation can make all the difference between a positive experience or a negative experience.

In fact, a 2017 GitHub survey that “Incomplete or Confusing Documentation” is the top complaint about open source software, TechRepublic writes “Ask a developer what her primary gripes with open source are, and documentation gets top billing, and by a wide margin.”

Technical writers across Google are addressing this issue, starting with open source projects in Google Cloud. We'll be sharing what we learn along the way and are excited to offer this brief guide as a starting point.

Open source software documentation

Providing effective documentation for open source software can build strong, inclusive communities and increase the usage of your product. The same factors that encourage collaborative development in open source projects can have the same positive results with documentation. Open source software provides unique ways to create effective documentation.

When software is open sourced, users are regarded as contributors and can access the source code and the documentation. They’re encouraged to submit additions, fix code, report bugs, and update documentation. Having more contributors can increase the rate at which software and documentation evolve.


The best way to accelerate software adoption is to describe its benefits and demonstrate how to use it. The benefits of timely, effective, and accurate content are numerous. Because documentation can save enough time and money to pay for itself, it:
  • Helps to create inclusive communities
  • Makes for a better software product
  • Promotes product adoption 
  • Reduces cost of ownership
  • Reduces the user’s learning curve
  • Makes for happy users
  • Improves the user interface
When documentation is inadequate, the opposite can occur. Incorrect, old, or missing documentation can:
  • Waste time
  • Cause errors and destroy data
  • Turn away customers
  • Increase support costs
  • Shorten a product’s life span

Why supplying effective documentation can be such a challenge

Useful documentation takes time to write and must be updated with the software. Did the installation process change? Are there better ways to configure performance? Was the user interface modified? Were new features introduced? Updates like these must be explained to new and existing customers.

It’s not enough to add words to a file and call it documentation.

How many times have you tried to use documentation that was five years out-of-date? How long did it take you to find another solution? (Not long, right?) Using old documentation can be like hiking an overgrown trail. The prospects of rogue branches, poison ivy, and getting lost suggest you are unlikely to emerge unscathed.

At first blush, writing documentation may seem optional. However, as a developer, you can literally be so close to your product that you take its features and purpose for granted. But your customers have no idea what you know, or how to apply what you know to address their business challenges. And time is money.

Remember the swimmer who got halfway across the English Channel only to say, “I’m tired,” before turning around and swimming back to shore? Ignoring the need for documentation is like that. Developing a software product gets you some of the way to your goal whereas helpful documentation fulfills your goal.

Momentum matters a lot. If you can settle into a rhythm of implementing new features, fixing bugs in the code, as well as writing and updating documentation, you can propel yourself to success.

Create an inclusive open source community

In the same GitHub survey referenced above, 95% of the respondents were men but evidence suggests that clear documentation:
  • Can contribute to inclusive communities
  • Is more highly valued by those groups who are typically under-represented
Documentation that effectively explains a project's processes, such as contributing to guides and defining codes of conduct, is valued more by groups that are underrepresented in the open source community, such as women.

Factors that can lead to ineffective documentation

Conditions like these almost always compromise the quality of documentation:
  • Belief that it’s enough for open source software to just work.
  • Notion that a specification is as good as instructional text.
  • Idea that casual unreviewed contributions are sufficient.
  • Belief that unscheduled and unspecified software updates are acceptable.
  • Minimal to no style guidelines.

Conquer your documentation challenges

Use these best practices to provide helpful and timely documentation:

1. Identify common terminology

Define and influence the usage and adoption of terminology for your open source project. Use the same terminology in the guides and in the product. Involving a writer early in product development can lead to a natural synergy between the user interface, guides, and training materials.

With clear definitions of terms and consistent usage in the documentation, you can teach your community to speak a common language. As a result, everyone in your products’ ecosystem can communicate more effectively with each other.

2. Provide contribution guidelines

Opensource.com describes exactly why contribution guidelines are so important. Consider how Kubernetes describes the types of documentation contributions your users can make and how to make them.

3. Create a documentation template

To make sure your contributors provide details in a consistent format, consider providing a document template to capture details for common topics such as:
  • Overview
  • Prerequisites
  • Procedural steps
  • What’s next

4. Document new and updated features

When a feature is added or updated, ask that it be documented. You can even provide guidelines to capture key information. Initially, some may think it cumbersome to require that  instructions be provided early in the development process. However, think of documentation to be like testing in that nobody really wants to do it but things work so much better. Sufficient testing and teaching are good for quality and momentum.

Code reviewers and maintainers of open source software have power. Code reviewers can (and should) withhold approval until documentation is sufficient.

Remember, not all changes require documentation updates. Here’s a good rule of thumb:

If an update to a project will require users to change their behavior, then documentation updates may be required.

If not, how will your customers find the new feature you worked so hard to implement? Said another way, if a change doesn't require tests, it probably doesn't require docs, either. Use your best judgement. For example, code refactoring and experimental tweaks need not be tested or documented.

As always, simplify and automate this process as much as possible. At Google, teams can enforce a presubmit check that either looks for a flag that indicates a doc update isn't necessary (presubmit checks for style issues can prevent a lot of arguments, too). We also allow owners of a file to submit changes without a review.

If your team balks at this requirement, remind them that simple project documentation is about sharing the information you have in your head so that many others can access it later without bothering you.

Documentation updates aren't typically onerous! The size of your documentation change scales with the size of your pull request (PR). If your PR contains a thousand lines of code, you may need to write a few hundred lines of documentation. If the PR contains a one-line change, you may need to change a word or two.

Finally, remember that documentation needn’t be perfect, but instead fit for use. What's most important is that key information is clearly conveyed.

5. Conduct regular freshness reviews

At least every quarter, review and update your content. Many hands make for many voices, many typos, and many inconsistencies. Don’t let content persist unchecked for years without periodically confirming it’s still useful.

In conclusion, success breeds success. By effectively documenting open source software, everybody wins.

We hope you'll put this guidance to work and help your open source project become even more successful! We'd love to hear from you if you do, or if you have questions or useful advice to share.

By Janet Davies, Google OpenDocs

Outline: secure access to the open web

Censorship and surveillance are challenges that many journalists around the world face on a daily basis. Some of them use a virtual private network (VPN) to provide safer access to the open internet, but not all VPNs are equally reliable and trustworthy, and even fewer are open source.

That’s why Jigsaw created Outline, a new open source, independently audited platform that lets any organization easily create and operate their own VPN.

Outline’s most striking feature is arguably how easy it is to use. An organization starts by downloading the Outline Manager app, which lets them sign in to DigitalOcean, where they can host their own VPN, and set it up with just a few clicks. They can also easily use other cloud providers, provided they have shell access to run the installation script. Once an Outline server is set up, the server administrator can create access credentials and share with their network of contacts, who can then use the Outline clients to connect to it.


A core element to any VPN’s security is the protocol that the server and clients use to communicate. When we looked at the existing protocols, we realized that many of them were easily identifiable by network adversaries looking to spot and block VPN traffic. To make Outline more resilient against this threat, we chose Shadowsocks, a secure, handshake-less, and open source protocol that is known for its strength and performance, and enjoys the support of many developers worldwide. Shadowsocks is a combination of a simplified SOCKS5-like routing protocol, running on top of an encrypted channel. We chose the AEAD_CHACHA20_POLY1305 cipher, which is an IETF standard and provides the security and performance users need.

Another important component to security is running up-to-date software. We package the server code as a Docker image, enabling us to run on multiple platforms, and allowing for automatic updates using Watchtower. On DigitalOcean installations, we also enable automatic security updates on the host machine.

If security is one of the most critical parts of creating a better VPN, usability is the other. We wanted Outline to offer a consistent, simple user experience across platforms, and for it to be easy for developers around the world to contribute to it. With that in mind, we use the cross-platform development framework Apache Cordova for Android, iOS, macOS and ChromeOS, and Electron for Windows. The application logic is a web application written in TypeScript, while the networking code had to be written in native code for each platform. This setup allows us to reutilize most of code, and create consistent user experiences across diverse platforms.

In order to encourage a robust developer community we wanted to strike a balance between simplicity, reproducibility, and automation of future contributions. To that end, we use Travis for continuous builds and to generate the binaries that are ultimately uploaded to the app stores. Thanks to its cross-platform support, any team member can produce a macOS or Windows binary with a single click. We also use Docker to package the build tools for client platforms, and thanks to Electron, developers familiar with the server's Node.js code base can also contribute to the Outline Manager application.

You can find our code in the Outline GitHub repositories and more information on the Outline website. We hope that more developers join the project to build technology that helps people connect to the open web and stay more safe online.

By Vinicius Fortuna, Jigsaw

Introducing the Kaggle “Quick, Draw!” Doodle Recognition Challenge



Online handwriting recognition consists of recognizing structured patterns in freeform handwritten input. While Google products like Translate, Keep and Handwriting Input use this technology to recognize handwritten text, it works for any predefined pattern for which enough training data is available. The same technology that lets you digitize handwritten text can also be used to improve your drawing abilities and build virtual worlds, and represents an exciting research direction that explores the potential of handwriting as a human-computer interaction modality. For example the “Quick, Draw!” game generated a dataset of 50M drawings (out of more than 1B that were drawn) which itself inspired many different new projects.

In order to encourage further research in this exciting field, we have launched the Kaggle "Quick, Draw!" Doodle Recognition Challenge, which tasks participants to build a better machine learning classifier for the existing “Quick, Draw!” dataset. Importantly, since the training data comes from the game itself (where drawings can be incomplete or may not match the label), this challenge requires the development of a classifier that can effectively learn from noisy data and perform well on a manually-labeled test set from a different distribution.

The Dataset
In the original “Quick, Draw!” game, the player is prompted to draw an image of a certain category (dog, cow, car, etc). The player then has 20 seconds to complete the drawing - if the computer recognizes the drawing correctly within that time, the player earns a point. Each game consists of 6 randomly chosen categories.
Because of the game mechanics, the labels in the Quick, Draw! dataset fall into the following categories:
  • Correct: the user drew the prompted category and the computer only recognized it correctly after the user was done drawing.
  • Correct, but incomplete: the user drew the prompted category and the computer recognized it correctly before the user had finished. Incompleteness can vary from nearly ready to only a fraction of the category drawn. This is probably fairly common in images that are marked as recognized correctly.
  • Correct, but not recognized correctly: The player drew the correct category but the AI never recognized it. Some players react to this by adding more details. Others scribble out and try again.
  • Incorrect: some players have different concepts in mind when they see a word - e.g. in the category seesaw, we have observed a number of saw drawings.
In addition to the labels described above, each drawing is given as a sequence of strokes, where each stroke is a sequence of touch points. While these can easily be rendered as images, using the order and direction of strokes often helps making handwriting recognizers better.

Get Started
We’ve previously published a tutorial that uses this dataset, and now we're inviting the community to build on this or other approaches to achieve even higher accuracy. You can get started by visiting the challenge website and going through the existing kernels which allow you to analyze the data and visualize it. We’re looking forward to learning about the approaches that the community comes up with for the competition and how much you can improve on our original production model.

Acknowledgements
We'd like to thank everyone who worked with us on this, particularly Jonas Jongejan and Brenda Fogg from the Creative Lab team, Julia Elliott and Walter Reade from the Kaggle team, and the handwriting recognition team.

Source: Google AI Blog


These 27 organizations will mentor students in Google Code-in 2018

We’re excited to welcome 27 open source organizations to mentor students as part of Google Code-in 2018. The contest, now in its ninth year, offers 13-17 year old pre-university students from around the world an opportunity to learn and practice their skills while contributing to open source projects–all online!

Google Code-in starts for students on October 23rd. Students are encouraged to learn about the participating organizations ahead of time and can get started by clicking on the links below:
  • AOSSIE: Australian umbrella organization for open source projects.
  • Apertium: rule-based machine translation platform.
  • Catrobat: visual programming for creating mobile games and animations.
  • CCExtractor: open source tools for subtitle generation.
  • CloudCV: building platforms for reproducible AI research.
  • coala: a unified interface for linting and fixing code, regardless of the programming languages used.
  • Copyleft Games Group: develops tools, libraries, and game engines.
  • Digital Impact Alliance: collaborative space for multiple open source projects serving the international development and humanitarian response sectors.
  • Drupal: content management platform.
  • Fedora Project: a free and friendly Linux-based operating system.
  • FOSSASIA: developing communities across all ages and borders to form a better future with Open Technologies and ICT.
  • Haiku: operating system specifically targeting personal computing.
  • JBoss Community: a community of projects around JBoss Middleware.
  • KDE Community: produces FOSS by artists, designers, programmers, translators, writers and other contributors.
  • Liquid Galaxy: an interactive, panoramic and immersive visualization tool.
  • MetaBrainz: builds community maintained databases.
  • MovingBlocks: a Minecraft-inspired open source game.
  • OpenMRS: open source medical records system for the world.
  • OpenWISP: build and manage low cost networks such as public wifi.
  • OSGeo: building open source geospatial tools.
  • PostgreSQL: relational database system.
  • Public Lab: open software to help communities measure and analyze pollution.
  • RTEMS Project: operating system used in satellites, particle accelerators, robots, racing motorcycles, building controls, medical devices.
  • Sugar Labs: learning platform and activities for elementary education.
  • SCoRe: research lab seeking sustainable solutions for problems faced by developing countries.
  • The ns-3 Network Simulator Project: packet-level network simulator for research and education.
  • Wikimedia: non-profit foundation dedicated to bringing free content to the world, operating Wikipedia.
These 27 organizations are hard at work creating thousands of tasks for students to work on, including code, documentation, design, quality assurance, outreach, research and training tasks. The contest starts for students on Tuesday, October 23rd at 9:00am Pacific Time.

You can learn more about Google Code-in on the contest site where you’ll find Frequently Asked Questions, Important Dates and flyers and other helpful information including the Getting Started Guide.

Want to talk with other students, mentors, and organization administrations about the contest? Check out our discussion mailing list. We can’t wait to get started!

By Stephanie Taylor, Google Open Source

Introducing the Unrestricted Adversarial Examples Challenge



Machine learning is being deployed in more and more real-world applications, including medicine, chemistry and agriculture. When it comes to deploying machine learning in safety-critical contexts, significant challenges remain. In particular, all known machine learning algorithms are vulnerable to adversarial examples — inputs that an attacker has intentionally designed to cause the model to make a mistake. While previous research on adversarial examples has mostly focused on investigating mistakes caused by small modifications in order to develop improved models, real-world adversarial agents are often not subject to the “small modification” constraint. Furthermore, machine learning algorithms can often make confident errors when faced with an adversary, which makes the development of classifiers that don’t make any confident mistakes, even in the presence of an adversary which can submit arbitrary inputs to try to fool the system, an important open problem.

Today we're announcing the Unrestricted Adversarial Examples Challenge, a community-based challenge to incentivize and measure progress towards the goal of zero confident classification errors in machine learning models. While previous research has focused on adversarial examples that are restricted to small changes to pre-labeled data points (allowing researchers to assume the image should have the same label after a small perturbation), this challenge allows unrestricted inputs, allowing participants to submit arbitrary images from the target classes to develop and test models on a wider variety of adversarial examples.
Adversarial examples can be generated through a variety of means, including by making small modifications to the input pixels, but also using spatial transformations, or simple guess-and-check to find misclassified inputs.
Structure of the Challenge
Participants can submit entries one of two roles: as a defender, by submitting a classifier which has been designed to be difficult to fool, or as an attacker, by submitting arbitrary inputs to try to fool the defenders' models. In a “warm-up” period before the challenge, we will present a set of fixed attacks for participants to design networks to defend against. After the community can conclusively beat those fixed attacks, we will launch the full two-sided challenge with prizes for both attacks and defenses.

For the purposes of this challenge, we have created a simple “bird-or-bicycle” classification task, where a classifier must answer the following: “Is this an unambiguous picture of a bird, a bicycle, or is it ambiguous / not obvious?” We selected this task because telling birds and bicycles apart is very easy for humans, but all known machine learning techniques struggle at the task when in the presence of an adversary.

The defender's goal is to correctly label a clean test set of birds and bicycles with high accuracy, while also making no confident errors on any attacker-provided bird or bicycle image. The attacker's goal is to find an image of a bird that the defending classifier confidently labels as a bicycle (or vice versa). We want to make the challenge as easy as possible for the defenders, so we discard all images that are ambiguous (such as a bird riding a bicycle) or not obvious (such as an aerial view of a park, or random noise).
Examples of ambiguous and unambiguous images. Defenders must make no confident mistakes on unambiguous bird or bicycle images. We discard all images that humans find ambiguous or not obvious. All images under CC licenses 1, 2, 3, 4.
Attackers may submit absolutely any image of a bird or a bicycle in an attempt to fool the defending classifier. For example, an attacker could take photographs of birds, use 3D rendering software, make image composites using image editing software, produce novel bird images with a generative model, or any other technique.

In order to validate new attacker-provided images, we ask an ensemble of humans to label the image. This procedure lets us allow attackers to submit arbitrary images, not just test set images modified in small ways. If the defending classifier confidently classifies as "bird" any attacker-provided image which the human labelers unanimously labeled as a bicycle, the defending model has been broken. You can learn more details about the structure of the challenge in our paper.

How to Participate
If you’re interested in participating, guidelines for getting started can be found on the project on github. We’ve already released our dataset, the evaluation pipeline, and baseline attacks for the warm-up, and we’ll be keeping an up-to-date leaderboard with the best defenses from the community. We look forward to your entries!

Acknowledgements
The team behind the Unrestricted Adversarial Examples Challenge includes Tom Brown, Catherine Olsson, Nicholas Carlini, Chiyuan Zhang, and Ian Goodfellow from Google, and Paul Christiano from OpenAI.

Source: Google AI Blog