Tag Archives: bazel

Google and Debian work together to make COVID-19 researchers’ lives easier

Posted by Joe Hicks, Yun Peng, Olek Wojnar

debian logo

Google and Debian work together to make COVID-19 researchers’ lives easier

  • Bazel is now available as an easy to install package distributed on Debian and Ubuntu.
  • Tensorflow packaging for Debian is progressing.

Olek Wojnar, Debian Developer, reached out to the Bazel team about packaging and distributing Bazel on Debian (and other Linux distributions such as Ubuntu) in service of delivering Tensorflow Machine Learning functionality for COVID-19 researchers:

“I'm working with the Debian Med team right now to get some much-needed software packaged and available for users in the medical community to help with the COVID-19 pandemic. At least one of the packages we desperately need requires Bazel to build. Clearly this is an unusual and very critical situation. I don't think it's an exaggeration to say that lives may literally depend on us getting better tools to the medical professionals out there, and quickly. The entire international community would be extraordinarily grateful if @google and the @bazelbuild team could prioritize helping with this!”

The Bazel team jumped in to help Olek and the COVID-19 research community. Yun Peng, Software Engineer at Google with Olek Wojnar led the team of Bazel and Debian volunteers to move the project forward. The joint effort between Debian and Google has produced some great results, including packaging the Bazel bootstrap variant in 6 months time (Debian 11 -- released in Late 2021; Ubuntu 21.04 -- 22 April 2021). Bazel is now available as an easy to install package distributed on Debian and Ubuntu. The extended Google team continues to work with Debian towards the next step of packaging and distributing Tensorflow on Debian and other Linux distributions.

In addition to Yun and Olek, other contributors to this project include Michael R. Crusoe of Debian, Joe Hicks, John Field, Philipp Wollermann, and Tobias Werth of Google.

Writing fuzz tests with ease using Bazel

We are announcing Bazel support for developing and testing fuzz tests, with OSS-Fuzz integration, through the new rules_fuzzing Bazel library.

Fuzzing is an effective, well-known testing technique for finding security and stability bugs in software. But writing and testing fuzz tests can be tedious. Developers typically need to:
  • Implement a fuzz driver function, which exercises the API under test;
  • Build the code with the proper instrumentation (such as Address Sanitizer);
  • Link it with one of the available fuzzing engine libraries (libFuzzer, AFL++, Honggfuzz, etc.) that provide the core test generation logic;
  • Run the fuzz test binary with the right set of flags (e.g., to specify corpora or dictionaries);
  • Package the fuzz test and its resources for consumption by fuzzing infrastructures, such as OSS-Fuzz.
Unfortunately, build systems don't traditionally offer any support beyond the core primitives of producing executables, so projects adopting fuzzing often end up reimplementing fuzz test recipes.

Bazel is a versatile and extensible build system, focused on scalable, reliable, and reproducible builds. Originally designed to scale to Google's entire monolithic repository, it now underpins large enterprises and key open source Internet infrastructure projects.

We are pleased to announce that projects using Bazel can get advanced fuzzing support through the new rules_fuzzing extension library. The new fuzzing rules take care of all the boilerplate needed to build and run fuzz tests. Developers simply write the fuzz driver code and define a build target for it (example driver and target for RE2). Fuzz tests can be built and run using a number of fuzzing engines provided out-of-the-box, such as libFuzzer and Honggfuzz, as well as sanitizers. The rule library also provides the ability to define additional fuzzing engines.

You can integrate the fuzzing library with around 10 LOC in your Bazel WORKSPACE file. Defining a fuzz test in Bazel is as easy as writing the following in your BUILD file:

load("@rules_fuzzing//fuzzing:cc_deps.bzl, "cc_fuzz_test")
cc_fuzz_test(
   name = "my_fuzz_test",
   srcs = ["my_fuzz_test.cc"],
   deps = [":my_library"],
)


You can easily test the fuzzer locally by invoking its launcher:

$ bazel run --config=asan-libfuzzer //:my_fuzz_test_run

To improve the effectiveness of test case generation, fuzz tests also support seed corpora and dictionaries, through additional rule attributes. They will automatically be validated and included in fuzz test runs. Fuzz tests also serve as regression tests on the seed corpus. For example, you can add previously found and fixed crashes to the corpus and have them tested in your CI workflows:

$ bazel test --config=asan-replay //:my_fuzz_test

The fuzzing rules provide built-in support for OSS-Fuzz, our continuous fuzzing service for open source projects. The OSS-Fuzz support drastically simplifies writing the build scripts in project integration by automatically packaging the fuzz test and its dependencies using the expected OSS-Fuzz structure.

The Envoy Proxy project is one of the early adopters of the fuzzing rules library. As a large, mature C++ codebase, Envoy has maintained its own custom implementation of fuzzing support for its over 50 fuzz targets written so far. By switching to the new Bazel fuzzing rules, Envoy's fuzz targets automatically gained new features, such as local running and testing tools and support for multiple fuzzing engines. At the same time, Envoy simplified its OSS-Fuzz integration scripts. Moreover, it will automatically gain future functionality (e.g., more effective fuzzing engines, better coverage tracking, improved corpus management) as the Bazel fuzzing rules library evolves.

The Bazel rules for fuzzing draw from Google's experience providing effective fuzzing tools to our internal developers. We hope the new Bazel support for fuzzing will lower the barrier to fuzzing adoption in open source communities, further increasing the security and reliability of many projects. To learn more about integrating the fuzzing rules into your project, take a look at the Getting Started section in the documentation.

By Stefan Bucur, Software Analysis, Asra Ali, Envoy, and Abhishek Arya, OSS-Fuzz – Google

Welcome Android Open Source Project (AOSP) to the Bazel ecosystem

After significant investment in understanding how best to build the Android Platform correctly and quickly, we are pleased to announce that the Android Platform is migrating from its current build systems (Soong and Make) to Bazel. While components of Bazel have been already checked into the Android Open Source Project (AOSP) source tree, this will be a phased migration over the next few Android releases which includes many concrete and digestible milestones to make the transformation as seamless and easy as possible. There will be no immediate impact to the Android Platform build workflow or the existing supported Android Platform Build tools in 2020 or 2021. Some of the changesto support Android Platform builds are already in Bazel, such as Bazel’s ability to parse and execute Ninja files to support a gradual migration.

Migrating to Bazel will enable AOSP to:
  • Provide more flexibility for configuring the AOSP build (better support for conditionals)
  • Allow for greater introspection into the AOSP build progress and dependencies
  • Enable correct and reproducible (hermetic) AOSP builds
  • Introduce a configuration mechanism that will reduce complexity of AOSP builds
  • Allow for greater integration of build and test activities
  • Combine all of these to drive significant improvements in build time and experience
The benefits of this migration to the Bazel community are:
  • Significant ongoing investment in Bazel to support Android Platform builds
  • Expansion of the Bazel ecosystem and community to include, initially, tens of thousands of Android Platform developers and Android handset OEMs and chipset vendors.
  • Google’s Bazel rules for building Android apps will be open sourced, used in AOSP, and maintained by Google in partnership with the Android / Bazel community
  • Better Bazel support for building Android Apps
  • Better rules support for other languages used to build Android Platform (Rust, Java, Python, Go, etc)
  • Strong support for Bazel Long Term Support (LTS) releases, which benefits the expanded Bazel community
  • Improved documentation (tutorials and reference)
The recent check-in of Bazel to AOSP begins an initial pilot phase, enabling Bazel to be used in place of Ninja as the execution engine to build AOSP. Bazel can also explore the AOSP build graph. We're pleased to be developing this functionality directly in the Bazel and AOSP codebases. As with most initial development efforts, this work is experimental in nature. Remember to use the currently supported Android Platform Build System for all production work.

We believe that these updates to the Android Platform Build System enable greater developer velocity, productivity, and happiness across the entire Android Platform ecosystem.

By Joe Hicks on behalf of the Bazel and AOSP infrastructure teams

Get ready for BazelCon 2020

With only 24 hours to go, BazelCon 2020 is shaping up to be a much anticipated gathering for the Bazel community and broader Build ecosystem. With over 1000 attendees, presentations by Googlers, as well as talks from industry Build leaders from Twitter, Dropbox, Uber, Pinterest, GrabTaxi, and more, we hope BazelCon 2020 will provide an opportunity for knowledge sharing, networking, and community building.

I am very excited by the keynote announcements, the migration stories at Twitter, Pinterest, and CarGurus, as well as technical deep dives on Bazel persistent workers, incompatible target skipping, migrating from Gradle to Bazel, and more. The “sold out” Birds of a Feather sessions and the Live Q&A with the Bazel team will bring the community together to discuss design docs, look at landings, and provide feedback on the direction of Bazel and the entire ecosystem.

We are also pleased to announce that, starting with the next major release (4.0), Bazel will support Long Term Support (LTS) releases as well as regular Rolling releases.

Some benefits of this new release cadence are:
  • Bazel will release stable, supported LTS releases on a predictable schedule with a long window without breaking changes
  • Bazel contributors / rules owners can prepare to support future LTS releases via rolling releases.
  • Bazel users can choose the release cadence that works best for them, since we will offer both LTS releases and rolling releases.
Long Term Support (LTS) releases:
  • We will create an LTS release every ~9 months => new LTS release branch, increment major version number.
  • Each LTS release will include all new features, bug fixes and (breaking) changes since the last major version.
  • Bazel will actively support each LTS branch for 9 months with critical bug fixes, but no new features.
  • Thereafter, Bazel will provide maintenance for two additional years with only security and OS compatibility fixes.
  • Bazel Federation reboot: Bazel will provide strong guidance about the ruleset versions that should be used with each Bazel release so that each user will not have to manage interoperability themselves.
Make sure that you register at http://goo.gle/bazelcon to be a part of the excitement of the premier build conference!

See you all at BazelCon 2020!

By Joe Hicks and the entire Bazel Team at Google

BazelCon 2019

Cross-posted from the original BazelCon 2019 recap .

Last month the Google Bazel team hosted its largest ever Bazel user conference: BazelCon 2019, an annual gathering of the community surrounding the Bazel build system. This is the main Bazel event of the year which serves as an opportunity for Bazel contributors, maintainers, and users to meet and learn from each other, present Bazel migration stories, educate new users, and collaborate together on the future of Bazel.

BazelCon 2019 by the Numbers

  • 400+ attendees (2x increase over BazelCon 2018)
  • 125 organizations represented including Microsoft, Spotify, Uber, Apple, Cruise, EA, Lyft, Tesla, SpaceX, SAP, Bloomberg, Wix, Etsy, BMW and others
  • 26 full-length talks and 15 lightning talks by members of the external community and Googlers
  • 16 hours of Q&A during Office Hours with Bazel team members
  • 45 Bazel Bootcamp attendees
  • 5 Birds of a Feather sessions on iOS, Python, Java, C++ and Front-end Bazel rules
  • 182 users in the #bazelcon2019 Slack channel

BazelCon 2019 Full Length Talks

The full playlist also includes lighting talks.
  • Keynote: The Role of Catastrophic Failure in Software Design – Jeff Atwood (Stack Overflow/Discourse)
  • Bazel State of the Union – John Field and Dmirty Lomov (Google)
  • Building Self Driving Cars with Bazel – Axel Uhlig and Patrick Ziegler (BMW Group)
  • Moving to a Bazel-based CI system: 6 Learnings – Or Shachar (Wix)
  • Bazel Federation – Florian Weikert (Google)
  • Lessons from our First 100,000 Bazel Builds – Kevin Gessner (Etsy)
  • Migrating Lyft-iOS to Bazel – Keith Smiley and Dave Lee (Lyft)
  • Test Selection – Benjamin Peterson (Dropbox)
  • Porting iOS Apps to Bazel – Oscar Bonilla (LinkedIn)
  • Boosting Dev Box Performance with Remote Execution for Non-Hermetic Build Engines – Erik Mavrinac (Microsoft)
  • Building on Key - Keeping your Actions and Remote Executions in Tune – George Gensure (UberATG)
  • Bazel remote execution API vs Goma – Mostyn Bramley-Moore (Vewd Software)
  • Integrating with ease: leveraging BuildStream interaction with Bazel build for consistent results – Daniel Silverstone (Codethink)
  • Building Self-Driving Cars with Bazel – Michael Broll and Nico Valigi (Cruise)
  • Make local development (with Bazel) great again! – Ittai Zeidman (Wix)
  • Gradle to Bazel – Chip Dickson and Charles Walker (SUM Global Technology)
  • Bazel Bootcamp – Kyle Cordes (Oasis Digital)
  • Bazel migration patterns: how to prove business value with a small investment – Alex Eagle and Greg Magolan (Google)
  • Dynamic scheduling: Fastest clean and incremental builds – Julio Merino (Google)
  • Building a great CI with Bazel – Philipp Wollermann (Google)
By Misha Narinsky, Bazel Team

Bazel Reaches 1.0 Milestone!

We're excited to announce the first General Availability release of Bazel, an open source build system designed to support a wide variety of programming languages and platforms.

Bazel was born of Google's own needs for highly scalable builds. When we open sourced Bazel back in 2015, we hoped that Bazel could fulfill similar needs in the software development industry. A growing list of Bazel users attests to the widespread demand for scalable, reproducible, and multi-lingual builds. Bazel helps Google be more open too: several large Google open source projects, such as Angular and TensorFlow, use Bazel. Users have reported 3x test time reductions and 10x faster build speeds after switching to Bazel.
With the 1.0 release we’re continuing to implement Bazel's vision:
  • Bazel builds are fast and correct. Every build and test run is incremental, on your developers’ machines and on your CI test system.
  • Bazel supports multi-language, multi-platform builds and tests. You can run a single command to build and test your entire source tree, no matter which combination of languages and platforms you target.
  • Bazel provides a uniform extension language, Starlark, to define builds for any language or platform.
  • Bazel works across all major development platforms (Linux, macOS, and Windows).
  • Bazel allows your builds to scale—it connects to distributed remote execution and caching services.
The key features of the 1.0 GA release are:
  • Semantic Versioning
Starting with Bazel 1.0, we will use semantic versioning for all Bazel releases. For example, all 1.x releases will be backwards-compatible with Bazel 1.0. We will have a window of at least three months between major (breaking) releases. We'll continue to publish minor releases of Bazel every month, cutting from GitHub HEAD.
  • Long-Term Support
Long-Term Support (LTS) releases give users confidence that the Bazel team has the capacity and the process to quickly and safely deliver fixes for critical bugs, including vulnerabilities.
  • Well-rounded features for Angular, Android, Java, and C++
The new features include end-to-end support for remote execution and caching, and support for standard package managers and third-party dependencies.
New to Bazel? Try the tutorial for your favorite language to get started.

With the 1.0 release we still have many exciting developments ahead of us. Follow our blog or Twitter account for regular updates. Feel free to contact us with questions or feedback on the mailing list, submit feature requests (and report bugs) in our GitHub issue tracker, and join our Slack channel. Finally, join us at the largest-ever BazelCon conference in December 2019 for an opportunity to meet other Bazel users and the Bazel team at Google, see demos and tech talks, and learn more about fast, correct, and large-scale builds.

Last but not least, we wouldn't have gotten here without the continued trust, support, encouragement, and feedback from the community of Bazel users and contributors. Heartfelt thanks to all of you from the Bazel team!

By Dmitry Lomov, Bazel Team

Building a Build System: Bazel reaches Beta

We're excited to announce the Beta release of Bazel, an open source build system designed to support a wide variety of different programming languages and platforms.

There are lots of other build systems out there -- Maven, Gradle, Ant, Make, and CMake just to name a few. So what’s special about Bazel? Bazel is what we use to build the large majority of software within Google. As such, it has been designed to handle build problems specific to Google’s development environment, including a massive, shared code repository in which all software is built from source, a heavy emphasis on automated testing and release processes, and language and platform diversity. Bazel isn’t right for every use case, but we believe that we’re not the only ones facing these kinds of problems and we want to contribute what we’ve learned so far to the larger developer community.

Our Beta release provides:


Check out the tutorial app to see a working example using several languages.

We still have a long way to go.  Looking ahead towards our 1.0.0 release, we plan to provide Windows support, distributed caching, and Go support among other features. See our roadmap for more details and follow our blog or Twitter account for regular updates.  Feel free to contact us with questions or feedback on the mailing list or IRC (#bazel on freenode).

By Jeff Cox, Bazel team