2022 was a successful year for Google's Vulnerability Reward Programs (VRPs), with over 2,900 security issues identified and fixed, and over $12 million in bounty rewards awarded to researchers. A significant amount of these vulnerability reports helped improve the security of Google Cloud products, which in turn helps improve security for our users, customers, and the Internet at large.

We first announced the Google Cloud VRP Prize in 2019 to encourage security researchers to focus on the security of Google Cloud and to incentivize sharing knowledge on Cloud vulnerability research with the world. This year, we were excited to see an increase in collaboration between researchers, which often led to more detailed and complex vulnerability reports. After careful evaluation of the submissions, today we are excited to announce the winners of the 2022 Google Cloud VRP Prize.

2022 Google Cloud VRP Prize Winners

1st Prize - $133,337: Yuval Avrahami for the report and write-up Privilege escalations in GKE Autopilot. Yuval's excellent write-up describes several attack paths that would allow an attacker with permission to create pods in an Autopilot cluster to escalate privileges and compromise the underlying node VMs. While these VMs are accessible to customers in GKE Standard, this research led to several hardening improvements in Autopilot that make it a better secure-by-default Kubernetes offering.

2nd Prize - $73,331: Sivanesh Ashok and Sreeram KL for the report and write-up SSH Key Injection on GCE. Their write-up describes the journey of discovering a vulnerability that would allow an attacker to gain access to a user's GCE VM by tricking them into clicking a link. They demonstrate the importance of persistence and turned a strange behavior in user creation into an injection of arbitrary SSH public keys.

3rd Prize -  $31,337: Sivanesh Ashok and Sreeram KL for the report and write-up Bypassing Authorization in Cloud Workstations. Their write-up describes their research process for analyzing Cloud Workstations and then a full-chain exploit to steal a user's access token by abusing the format of an OAuth state parameter.

4th Prize - $31,311: Sreeram KL and Sivanesh Ashok for the report and write-up Client-Side SSRF to Google Cloud Project Takeover. Their write-up combines a client-side SSRF, a CSRF bypass, and a clever 3xx redirect by "deactivating" a Feedburner proxy. An attacker could use this vulnerability to steal a Vertex AI user's access token by tricking them into clicking a link.

5th Prize - $17,311: Yuval Avrahami and Shaul Ben Hai for the report and write-up Kubernetes Privilege Escalation: Excessive Permissions in Popular Platforms. Their whitepaper covers privilege escalation vectors in Kubernetes and describes vulnerabilities in many Kubernetes hosting providers, including Azure's AKS, Amazon's EKS, and GKE.

6th Prize - $13,373: Obmi for the report and write-up A Few Bugs in the Google Cloud Shell. Obmi discovered vulnerabilities in the Cloud Shell file upload functionality that would have allowed an attacker to write arbitrary files to a user's Cloud Shell via cross-site request forgery.

7th Prize - $13,337: Bugra Eskici for the report and write-up Command injection in Cloud Shell. Bugra found a very curious injection point in a Cloud Shell script that led to a URL query parameter being directly injected into a Python script. This vulnerability would have given an attacker arbitrary code execution in a user's Cloud Shell if they clicked on an attacker-controlled link.

Every day, personal data, such as location information, images, or text queries are passed between your device and remote, cloud-based services. Your data is encrypted when in transit and at rest, but as potential attack vectors grow more sophisticated, data must also be protected during use by the service, especially for software systems that handle personally identifiable user data.

Toward this goal, Google’s Project Oak is a research effort that relies on the confidential computing paradigm to build an infrastructure for processing sensitive user data in a secure and privacy-preserving way: we ensure data is protected during transit, at rest, and while in use. As an assurance that the user data is in fact protected, we’ve open sourced Project Oak code, and have introduced a transparent release process to provide publicly inspectable evidence that the application was built from that source code. 

This blog post introduces Oak's transparent release process, which relies on the SLSA framework to generate cryptographic proof of the origin of Oak’s confidential computing stack, and together with Oak’s remote attestation process, allows users to cryptographically verify that their personal data was processed by a trustworthy application in a secure environment. 

Integrity and transparency with SLSA  

Project Oak recently collaborated with the SLSA community to create a new container-based builder that produces Supply-chain Levels for Software Artifacts (SLSA) Build Level 3 provenance. This new container-based builder generates non-forgeable provenance statements that capture details about build process information that allow users to perform automated, rigorous provenance verification.

With this enhanced provenance generated by the container-based builder, you can answer questions like:

• Was the artifact built with a toolchain that I know and trust?

• Was the artifact built with a command that I trust?

• Did the build command use a tool that was affected by a vulnerability?

• How can I recreate the artifact?

Project Oak is particularly interested in answering these questions about every layer of the confidential computing stack. For instance, to be sure that a released binary was built using a trusted build process (e.g., the build command did not use any potentially malicious tool), the Oak release process compares the build command against a set of allow-listed tokens. Similarly, we can verify that the builder was not tampered with. 

Transparent releases: an added layer of trust

Project Oak develops a secure runtime and a remote attestation protocol—ways to detect potential adversaries on remote servers and to protect workloads while they are running. Now, with the addition of the container-based SLSA builder, we are able to complete our transparent release process to protect against software supply chain attacks and provide an automated process for verifying the integrity and trustworthiness of a remote server, before sending sensitive information to it. 

Specifically, for each released version of the Oak secure runtime, the Oak team generates and signs an endorsement statement for the binary, using a key accessible only to the Oak team. The endorsement statement can only be generated if the provenance statement passes verification checks, ensuring that a potential malicious attacker cannot forge the statement.

When the client establishes a connection to the server, the client must verify the endorsement statement and the proof of its inclusion in a transparency log, and check that the binary identities in the attestation report and the endorsement statement are the same. This, together with signature verification for the endorsement statement, guarantees three important points of trust for the overall process: that the client is interacting with the same publicly endorsed version of the Oak secure runtime that all other clients interact with; the Oak secure runtime is open source; and that it has a publicly published non-forgeable SLSA v1.0 provenance with adherence to SLSA Build Track 3. For a more technical explanation of the process, see Project Oak’s transparent release process.

Visualization of an Oak application, with attestation verification 

Try it out

We encourage you to check out the transparent release project as a use case for SLSA. Please reach out to us via our slack channel to explore ideas related to Oak secure runtimes and remote attestation.

You don’t need to use Project Oak to take advantage of the new SLSA builder tool. If your project is open source, try one of the SLSA builders to generate non-forgeable provenance for your binaries. We encourage you to containerize your build and try the container-based SLSA 3 builder! Using a container image for your builds improves the reproducibility of your binaries. We also recommend adding the instructions for building your container image (e.g., a Dockerfile) to your GitHub repository, which improves auditability and transparency of your build process, and thus the security of your software supply chain.

In 2020, we integrated kCTF into Google's Vulnerability Rewards Program (VRP) to support researchers evaluating the security of Google Kubernetes Engine (GKE) and the underlying Linux kernel. As the Linux kernel is a key component not just for Google, but for the Internet, we started heavily investing in this area. We extended the VRP's scope and maximum reward in 2021 (to $50k), then again in February 2022 (to $91k), and finally in August 2022 (to $133k). In 2022, we also summarized our learnings to date in our cookbook, and introduced our experimental mitigations for the most common exploitation techniques.

In this post, we'd like to share our learnings and statistics about the latest Linux kernel exploit submissions, how effective our mitigations are against them, what we do to protect our users, and, finally, how we are changing our program to align incentives to the areas we are most interested in.

Learnings and Statistics

Since its inception, the program has rewarded researchers with a total of 1.8 million USD, and in the past year, there has been a clear trend: 60% of the submissions exploited the io_uring component of the Linux kernel (we paid out around 1 million USD for io_uring alone). Furthermore, io_uring vulnerabilities were used in all the submissions which bypassed our mitigations.

Limiting io_uring

To protect our users, we decided to limit the usage of io_uring in Google products: 

• ChromeOS: We disabled io_uring (while we explore new ways to sandbox it).

• Android: Our seccomp-bpf filter ensures that io_uring is unreachable to apps. Future Android releases will use SELinux to limit io_uring access to a select few system processes. 

• GKE AutoPilot: We are investigating disabling io_uring by default.

• It is disabled on production Google servers.

While io_uring brings performance benefits, and promptly reacts to security issues with comprehensive security fixes (like backporting the 5.15 version to the 5.10 stable tree), it is a fairly new part of the kernel. As such, io_uring continues to be actively developed, but it is still affected by severe vulnerabilities and also provides strong exploitation primitives. For these reasons, we currently consider it safe only for use by trusted components.

Transparency

Currently, we make vulnerability details public on our spreadsheet (which now also includes CVE details), and we have summarized different exploitation techniques in our cookbook. In the future, to make our efforts more transparent and give faster feedback to the community, we will ask researchers to open-source their submissions, including the code they used.

Introducing kernelCTF

To better align incentives with our areas of interest, we are shifting our focus from GKE and kCTF to the latest stable kernel and our mitigations. As a result, starting today we will handle kernel exploit submissions under a new name, "kernelCTF," with its own reward structure and submission process. The maximum total payout for kernelCTF is still $133,337 per submission. While the specific GKE kernel configuration is still covered by the new kernelCTF, exploits affecting non-kernel components like the full GKE stack (including Kubernetes), the container runtime, and GKE itself, are now separately eligible for vulnerability rewards under the kCTF VRP which is returning to its original reward amounts and conditions.

Conclusion

Our goal remains the same: we are building a pipeline to analyze, experiment, measure, and build security mitigations to make the Linux kernel as safe as possible, with the help of the security community. We hope that over time, we will be able to implement security mitigations that make it more difficult to exploit Linux kernel vulnerabilities.

It’s Google CTF time! Get your hacking toolbox ready and prepare your caffeine for rapid intake. The competition kicks off on June 23 2023 6:00 PM UTC and runs through June 25 2023 6:00 PM UTC. Registration is now open at g.co/ctf.

Google CTF gives you a chance to challenge your skillz, show off your hacktastic abilities, and learn some new tricks along the way. It consists of a set of computer security puzzles (or challenges) involving reverse-engineering, memory corruption, cryptography, web technologies, and more. Use obscure security knowledge to find exploits through bugs and creative misuse. With each completed challenge your team will earn points and move up through the ranks. 

The top 8 teams will qualify for our Hackceler8 competition taking place in Tokyo later this year. Hackceler8 is our experimental esport-style hacking game, custom-made to mix CTF and speedrunning. In the competition, teams need to find clever ways to abuse the game features to capture flags as quickly as possible. See the 2022 highlight reel to get a sense of what it’s like. The prize pool for this year’s event stands at more than $32,000!

Screenshot from Hackeler8 2022 speedrun competition

Itching to get started early? Want to learn more, or get a leg up on the competition? Review challenges from previous years, including previous Hackceler8 matches, all open-sourced here. Or gain inspiration by binge watching hours of Hackceler8 2020 videos!

If you are just starting out in this space, check out last year’s event H4CK1NG GOOGLE! It’s a great way to get acquainted with security. You can also get ready for this year’s Beginner’s Quest that’ll be launching later this summer which will be in the theme of Computer History, so get ready for some technology archaeology.

Whether you’re a seasoned CTF player or just curious about cyber security and ethical hacking, we want you to join us. Sign up to expand your skill set, meet new friends in the security community, and even watch the pros in action. For the latest announcements, see g.co/ctf, subscribe to our mailing list, or follow us on Twitter @GoogleVRP. Interested in bug hunting for Google? Check out bughunters.google.com. See you there!

Today, we are announcing the launch of the v0.1 version of Graph for Understanding Artifact Composition (GUAC). Introduced at Kubecon 2022 in October, GUAC targets a critical need in the software industry to understand the software supply chain. In collaboration with Kusari, Purdue University, Citi, and community members, we have incorporated feedback from our early testers to improve GUAC and make it more useful for security professionals. This improved version is now available as an API for you to start developing on top of, and integrating into, your systems.

The need for GUAC

High-profile incidents such as Solarwinds, and the recent 3CX supply chain double-exposure, are evidence that supply chain attacks are getting more sophisticated. As highlighted by the U.S. Executive Order on Cybersecurity, there’s a critical need for security professionals, CISOs, and security engineers to be able to more deeply link information from different supply chain ecosystems to keep up with attackers and prevent exposure. Without linking different sources of information, it’s impossible to have a clear understanding of the potential risks posed by the software components in an organization. 

GUAC aggregates software security metadata and maps it to a standard vocabulary of concepts relevant to the software supply chain. This data can be accessed via a GraphQL interface, allowing development of a rich ecosystem of integrations, command-line tools, visualizations, and policy engines. 

We hope that GUAC will help the wider software development community better evaluate the supply chain security posture of their organizations and projects. Feedback from early adopters has been overwhelmingly positive: 

“At Yahoo, we have found immense value and significant efficiency by utilizing the open source project GUAC. GUAC has allowed us to streamline our processes and increase efficiency in a way that was not possible before,” said Hemil Kadakia, Sr. Mgr. Software Dev Engineering, Paranoids, Yahoo.

The power of GUAC

Dynamic aggregation

GUAC is not just a static database—it is the first application that is continuously evolving the database pertaining to the software that an organization develops or uses. Supply chains change daily, and by aggregating your Software Bill of Materials (SBOMs) and Supply-chain Levels for Software Artifacts (SLSA) attestations with threat intelligence sources (e.g., OSV vulnerability feeds) and OSS insights (e.g., deps.dev), GUAC is constantly incorporating the latest threat information and deeper analytics to help paint a more complete picture of your risk profile. And by merging external data with internal private metadata, GUAC brings the same level of reasoning to a company’s first-party software portfolio.

Seamless integration of incomplete metadata

Because of the complexity of the modern software stack—often spanning languages and toolchains—we discovered during GUAC development that it is difficult to produce high-quality SBOMs that are accurate, complete, and meet specifications and intents. 

Following the U.S. Executive Order on Cybersecurity, there are now a large number of SBOM documents being generated during release and build workflows to explain to consumers what’s in their software. Given the difficulty in producing accurate SBOMs, consumers often face a situation where they have incomplete, inaccurate, or conflicting SBOMs. In these situations, GUAC can fill in the gaps in the various supply chain metadata: GUAC can link the documents and then use heuristics to improve the quality of data and guess at the correct intent. Additionally, the GUAC community is now working closely with SPDX to advance SBOM tooling and improve the quality of metadata. 

GUAC's process for incorporating and enriching metadata for organizational insight

Consistent interfaces

Alongside the boom in SBOM production, there’s been a rapid expansion of new standards, document types, and formats, making it hard to perform consistent queries. The multiple formats for software supply chain metadata often refer to similar concepts, but with different terms. To integrate these, GUAC defines a common vocabulary for talking about the software supply chain—for example, artifacts, packages, repositories, and the relationships between them. 

This vocabulary is then exposed as a GraphQL API, empowering users to build powerful integrations on top of GUAC’s knowledge graph. For example, users are able to query seamlessly with the same commands across different SBOM formats like SPDX and CycloneDX. 

According to Ed Warnicke, Distinguished Engineer at Cisco Systems, "Supply chain security is increasingly about making sense of many different kinds of metadata from many different sources. GUAC knits all of that information together into something understandable and actionable." 

Potential integrations

Based on these features, we envision potential integrations that users can build on top of GUAC in order to:

• Create policies based on trust

• Quickly react to security compromises 

• Determine an upgrade plan in response to a security incident

• Create visualizers for data explorations, CLI tools for large scale analysis and incident response, CI checks, IDE plugins to shift policy left, and more

Developers can also build data source integrations under GUAC to expand its coverage. The entire GUAC architecture is plug-and-play, so you can write data integrations to get:

• Supply chain metadata from new sources like your preferred security vendors

• Parsers to translate this metadata into the GUAC ontology

• Database backends to store the GUAC data in either common databases or in organization-defined private data stores

GUAC's GraphQL query API enables a diverse ecosystem of tooling

Dejan Bosanac, an engineer at Red Hat and an active contributor to the GUAC project, further described GUAC’s ingestion abilities, “With mechanisms to ingest and certify data from various sources and GraphQL API to later query those data, we see it as a good foundation for our current and future SSCS efforts. Being a true open source initiative with a welcoming community is just a plus.” 

Next steps

Google is committed to making GUAC the best metadata synthesis and aggregation tool for security professionals. GUAC contributors are excited to meet at our monthly community calls and look forward to seeing demos of new applications built with GUAC.

“At Kusari, we are proud to have joined forces with Google's Open Source Security Team and the community to create and build GUAC,” says Tim Miller, CEO of Kusari. “With GUAC, we believe in the critical role it plays in safeguarding the software supply chain and we are dedicated to ensuring its success in the ecosystem.” 

Google is preparing SBOMs for consumption by the US Federal Government following EO 14028, and we are internally ingesting our SBOM catalog into GUAC to gather early insights. We encourage you to do the same with the GUAC release and submit your feedback. If the API is not flexible enough, please let us know how we can extend it. You can also submit suggestions and feedback on GUAC development or use cases, either by emailing [email protected] or filing an issue on our GitHub repository.

We hope you'll join us in this journey with GUAC!

Google’s Open Source Security Team recently sponsored a fuzzing competition as part of ISCE’s Search-Based and Fuzz Testing (SBFT) Workshop. Our goal was to encourage the development of new fuzzing techniques, which can lead to the discovery of software vulnerabilities and ultimately a safer open source ecosystem. 

The competitors’ fuzzers were judged on code coverage and their ability to discover bugs: 

• HasteFuzz took the $11,337 prize for code coverage

• PASTIS and AFLrustrust tied for bug discovery and split the $11,337 prize

Competitors were evaluated using FuzzBench, Google’s open source platform for testing and comparing fuzzers. The platform boasts a wide range of real world benchmarks and vulnerabilities, allowing researchers to test their fuzzers in an authentic environment. We hope the results of the SBFT fuzzing competition will lead to more efficient fuzzers and eventually newly discovered vulnerabilities. 

A closer look at our winners

Eight teams submitted fuzzers to the final competition and an additional four industry fuzzers (AFL++, libFuzzer, Honggfuzz, and AFL) were included as controls to represent current practice. 

• Code coverage winner - HasteFuzz

HasteFuzz, is a modification of the widely used AFL++ fuzzer. HasteFuzz filters out potentially duplicate inputs to increase efficiency, making it able to cover more code in the 23-hour test window because it is not likely to be retracing its steps. AFL++ is already a strong fuzzer—it had the best code coverage of the industry fuzzers tested in this competition—and HasteFuzz’s filtering took it to the next level.

• Bug discovery winners - PASTIS and AFLrustrust

PASTIS makes use of multiple fuzzing engines that can independently cover different program locations, allowing PASTIS to find bugs quickly. AFLrustrust rewrites AFL++ on top of LibAFL, which is a library of features that allows you to customize existing fuzzers. AFLrustrust effectively prunes redundant test cases, improving its bug finding efficiency. Both PASTIS and AFLrustrust found 8 out of 15 possible bugs, with each fuzzer missing only one bug discovered by others. They both outperformed the industry fuzzers, which found 7 or fewer bugs under the same constraints.

Additional competitors, such as AFL+++ and AFLSmart++, also showed improvements over the industry controls, a result we had hoped for with the competition.

Fuzzing research continues

The innovation and improvement shown through the SBFT fuzzing competition is one example of why we have invested in the FuzzBench project. Since its launch in 2020, FuzzBench has significantly contributed to high-quality fuzzing research, conducting over 900 experiments and discussed in more than 100 academic papers. FuzzBench was provided as a resource for the SBFT competition, but it is also available to researchers every day as a service. If you are interested in testing your fuzzers on FuzzBench, please see our guide to adding your fuzzer.

FuzzBench is in active development. We’d welcome feedback from any current or prospective FuzzBench users, your responses to this survey can help us plan the future of FuzzBench.

The Google Open Source Security Team would like to thank the ISCE conference and the SBFT workshop for hosting the fuzzing competition. We also want to thank each participant for their hard work. Together, we continue to push the boundaries of software security and create a safer, more robust open source ecosystem.