Author Archives: Sarah O'Rourke

Privileged pod escalations in Kubernetes and GKE

At the KubeCon EU 2022 conference in Valencia, security researchers from Palo Alto Networks presented research findings on “trampoline pods”—pods with an elevated set of privileges required to do their job, but that could conceivably be used as a jumping off point to gain escalated privileges.

The research mentions GKE, including how developers should look at the privileged pod problem today, what the GKE team is doing to minimize the use of privileged pods, and actions GKE users can take to protect their clusters.

Privileged pods within the context of GKE security

While privileged pods can pose a security issue, it’s important to look at them within the overall context of GKE security. To use a privileged pod as a “trampoline” in GKE, there is a major prerequisite – the attacker has to first execute a successful application compromise and container breakout attack.

Because the use of privileged pods in an attack requires a first step such as a container breakout to be effective, let’s look at two areas:
  1. features of GKE you can use to reduce the likelihood of a container breakout
  2. steps the GKE team is taking to minimize the use of privileged pods and the privileges needed in them.
Reducing container breakouts

There are a number of features in GKE along with some best practices that you can use to reduce the likelihood of a container breakout:

More information can be found in the GKE Hardening Guide.

How GKE is reducing the use of privileged pods.

While it’s not uncommon for customers to install privileged pods into their clusters, GKE works to minimize the privilege levels held by our system components, especially those that are enabled by default. However, there are limits as to how many privileges can be removed from certain features. For example, Anthos Config Management requires permissions to modify most Kubernetes objects to be able to create and manage those objects.

Some other privileges are baked into the system, such as those held by Kubelet. Previously, we worked with the Kubernetes community to build the Node Restriction and Node Authorizer features to limit Kubelet's access to highly sensitive objects, such as secrets, adding protection against an attacker with access to the Kubelet credentials.

More recently, we have taken steps to reduce the number of privileged pods across GKE and have added additional documentation on privileges used in system pods as well as information on how to improve pod isolation. Below are the steps we’ve taken:
  1. We have added an admission controller to GKE Autopilot and GKE Standard (on by default) and GKE/Anthos (opt-in) that stops attempts to run as a more privileged service account, which blocks a method of escalating privileges using privileged pods.
  2. We created a permission scanning tool that identifies pods that have privileges that could be used for escalation, and we used that tool to perform an audit across GKE and Anthos.
  3. The permission scanning tool is now integrated into our standard code review and testing processes to reduce the risk of introducing privileged pods into the system. As mentioned earlier, some features require privileges to perform their function.
  4. We are using the audit results to reduce permissions available to pods. For example, we removed “update nodes and pods” permissions from anetd in GKE.
  5. Where privileged pods are required for the operation of a feature, we’ve added additional documentation to illustrate that fact.
  6. We added documentation that outlines how to isolate GKE-managed workloads in dedicated node pools when you’re unable to use GKE Sandbox to reduce the risk of privilege escalation attacks.
In addition to the measures above, we recommend users take advantage of tools that can scan RBAC settings to detect overprivileged pods used in their applications. As part of their presentation, the Palo Alto researchers announced an open source tool, called rbac-police, that can be used for the task. So, while it only takes a single overprivileged workload to trampoline to the cluster, there are a number of actions you can take to minimize the likelihood of the prerequisite container breakout and the number of privileges used by a pod.

Announcing the winners of the 2020 GCP VRP Prize

We first announced the GCP VRP Prize in 2019 to encourage security researchers to focus on the security of Google Cloud Platform (GCP), in turn helping us make GCP more secure for our users, customers, and the internet at large. In the first iteration of the prize, we awarded $100,000 to the winning write-up about a security vulnerability in GCP. We also announced that we would reward the top 6 submissions in 2020 and increased the total prize money to $313,337.

2020 turned out to be an amazing year for the Google Vulnerability Reward Program. We received many high-quality vulnerability reports from our talented and prolific vulnerability researchers.

Vulnerability reports received year-over-year

This trend was reflected in the submissions we received for the GCP VRP Prize. After careful evaluation of the many innovative and high-impact vulnerability write-ups we received this year, we are excited to announce the winners of the 2020 GCP VRP Prize:
  • First Prize, $133,337: Ezequiel Pereira for the report and write-up RCE in Google Cloud Deployment Manager. The bug discovered by Ezequiel allowed him to make requests to internal Google services, authenticated as a privileged service account. Here's a video that gives more details about the bug and the discovery process.

  • Second Prize, $73,331: David Nechuta for the report and write-up 31k$ SSRF in Google Cloud Monitoring led to metadata exposure. David found a Server-side Request Forgery (SSRF) bug in Google Cloud Monitoring's uptime check feature. The bug could have been used to leak the authentication token of the service account used for these checks.
  • Third Prize, $73,331: Dylan Ayrey and Allison Donovan for the report and write-up Fixing a Google Vulnerability. They pointed out issues in the default permissions associated with some of the service accounts used by GCP services.
  • Fourth Prize, $31,337: Bastien Chatelard for the report and write-up Escaping GKE gVisor sandboxing using metadata. Bastien discovered a bug in the GKE gVisor sandbox's network policy implementation due to which the Google Compute Engine metadata API was accessible. 
  • Fifth Prize, $1,001: Brad Geesaman for the report and write-up CVE-2020-15157 "ContainerDrip" Write-up. The bug could allow an attacker to trick containerd into leaking instance metadata by supplying a malicious container image manifest.
  • Sixth Prize, $1,000: Chris Moberly for the report and write-up Privilege Escalation in Google Cloud Platform's OS Login. The report demonstrates how an attacker can use DHCP poisoning to escalate their privileges on a Google Compute Engine VM.
Congratulations to all the winners! If we have peaked your interest and you would like to enter the competition for a GCP VRP Prize in 2021, here’s a reminder on the requirements.
  • Find a vulnerability in a GCP product (check out Google Cloud Free Program to get started)
  • Report it to the VRP (you might get rewarded for it on top of the GCP VRP Prize!)
  • Create a public write-up
  • Submit it here
Make sure to submit your VRP reports and write-ups before December 31, 2021 at 11:59 GMT. Good luck!You can learn more about the prize for this year here. We can't wait to see what our talented vulnerability researchers come up with this year!

Launching OSV – Better vulnerability triage for open source

Know, Prevent, Fix: A framework for shifting the discussion around vulnerabilities in open source

Executive Summary:

The security of open source software has rightfully garnered the industry’s attention, but solutions require consensus about the challenges and cooperation in the execution. The problem is complex and there are many facets to cover: supply chain, dependency management, identity, and build pipelines. Solutions come faster when the problem is well-framed; we propose a framework (“Know, Prevent, Fix”) for how the industry can think about vulnerabilities in open source and concrete areas to address first, including:

  • Consensus on metadata and identity standards: We need consensus on fundamentals to tackle these complex problems as an industry. Agreements on metadata details and identities will enable automation, reduce the effort required to update software, and minimize the impact of vulnerabilities.
  • Increased transparency and review for critical software: For software that is critical to security, we need to agree on development processes that ensure sufficient review, avoid unilateral changes, and transparently lead to well-defined, verifiable official versions.

The following framework and goals are proposed with the intention of sparking industry-wide discussion and progress on the security of open source software.

Due to recent events, the software world gained a deeper understanding about the real risk of supply-chain attacks. Open source software should be less risky on the security front, as all of the code and dependencies are in the open and available for inspection and verification. And while that is generally true, it assumes people are actually looking. With so many dependencies, it is impractical to monitor them all, and many open source packages are not well maintained.

It is common for a program to depend, directly or indirectly, on thousands of packages and libraries. For example, Kubernetes now depends on about 1,000 packages. Open source likely makes more use of dependencies than closed source, and from a wider range of suppliers; the number of distinct entities that need to be trusted can be very high. This makes it extremely difficult to understand how open source is used in products and what vulnerabilities might be relevant. There is also no assurance that what is built matches the source code.

Taking a step back, although supply-chain attacks are a risk, the vast majority of vulnerabilities are mundane and unintentional—honest errors made by well-intentioned developers. Furthermore, bad actors are more likely to exploit known vulnerabilities than to find their own: it’s just easier. As such, we must focus on making fundamental changes to address the majority of vulnerabilities, as doing so will move the entire industry far along in addressing the complex cases as well, including supply-chain attacks.

Few organizations can verify all of the packages they use, let alone all of the updates to those packages. In the current landscape, tracking these packages takes a non-trivial amount of infrastructure, and significant manual effort. At Google, we have those resources and go to extraordinary lengths to manage the open source packages we use—including keeping a private repo of all open source packages we use internally—and it is still challenging to track all of the updates. The sheer flow of updates is daunting. A core part of any solution will be more automation, and this will be a key theme for our open source security work in 2021 and beyond.

Because this is a complex problem that needs industry cooperation, our purpose here is to focus the conversation around concrete goals. Google co-founded the OpenSSF to be a focal point for this collaboration, but to make progress, we need participation across the industry, and agreement on what the problems are and how we might address them. To get the discussion started, we present one way to frame this problem, and a set of concrete goals that we hope will accelerate industry-wide solutions.

We suggest framing the challenge as three largely independent problem areas, each with concrete objectives:

  1. Know about the vulnerabilities in your software
  2. Prevent the addition of new vulnerabilities, and
  3. Fix or remove vulnerabilities.

A related but separate problem, which is critical to securing the supply chain, is improving the security of the development process. We’ve outlined the challenges of this problem and proposed goals in the fourth section, Prevention for Critical Software.

Know your Vulnerabilities

Knowing your vulnerabilities is harder than expected for many reasons. Although there are mechanisms for reporting vulnerabilities, it is hard to know if they actually affect the specific versions of software you are using.

Goal: Precise Vulnerability Data

First, it is crucial to capture precise vulnerability metadata from all available data sources. For example, knowing which version introduced a vulnerability helps determine if one's software is affected, and knowing when it was fixed results in accurate and timely patching (and a reduced window for potential exploitation). Ideally, this triaging workflow should be automated.

Second, most vulnerabilities are in your dependencies, rather than the code you write or control directly. Thus, even when your code is not changing, there can be a constant churn in your vulnerabilities: some get fixed and others get added.1

Goal: Standard Schema for Vulnerability Databases

Infrastructure and industry standards are needed to track and maintain open source vulnerabilities, understand their consequences, and manage their mitigations. A standard vulnerability schema would allow common tools to work across multiple vulnerability databases and simplify the task of tracking, especially when vulnerabilities touch multiple languages or subsystems.

Goal: Accurate Tracking of Dependencies

Better tooling is needed to understand quickly what software is affected by a newly discovered vulnerability, a problem made harder by the scale and dynamic nature of large dependency trees. Current practices also often make it difficult to predict exactly what versions are used without actually doing an installation, as the software for version resolution is only available through the installer.

Prevent New Vulnerabilities

It would be ideal to prevent vulnerabilities from ever being created, and although testing and analysis tools can help, prevention will always be a hard problem. Here we focus on two specific aspects:

  • Understanding risks when deciding on a new dependency
  • Improving development processes for critical software

Goal: Understand the Risks for New Dependencies

The first category is essentially knowing about vulnerabilities at the time you decide to use a package. Taking on a new dependency has inherent risk and it needs to be an informed decision. Once you have a dependency, it generally becomes harder to remove over time.

Knowing about vulnerabilities is a great start, but there is more that we can do.

Many vulnerabilities arise from lack of adherence to security best practices in software development processes. Are all contributors using two-factor authentication (2FA)? Does the project have continuous integration set up and running tests? Is fuzzing integrated? These are the types of security checks that would help consumers understand the risks they’re taking on with new dependencies. Packages with a low “score” warrant a closer review, and a plan for remediation.

The recently announced Security Scorecards project from OpenSSF attempts to generate these data points in a fully automated way. Using scorecards can also help defend against prevalent typosquatting attacks (malevolent packages with names similar to popular packages), since they would score much lower and fail many security checks.

Improving the development processes for critical software is related to vulnerability prevention, but deserves its own discussion further down in our post.

Fix or Remove Vulnerabilities

The general problem of fixing vulnerabilities is beyond our scope, but there is much we can do for the specific problem of managing vulnerabilities in software dependencies. Today there is little help on this front, but as we improve precision it becomes worthwhile to invest in new processes and tooling.

One option of course is to fix the vulnerability directly. If you can do this in a backwards-compatible way, then the fix is available for everyone. But a challenge is that you are unlikely to have expertise on the problem, nor the direct ability to make changes. Fixing a vulnerability also assumes the software maintainers are aware of the issue, and have the knowledge and resources for vulnerability disclosure.

Conversely, if you simply remove the dependency that contains the vulnerability, then it is fixed for you and those that import or use your software, but not for anyone else. This is a change that is under your direct control.

These scenarios represent the two ends of the chain of dependencies between your software and the vulnerability, but in practice there can be many intervening packages. The general hope is that someone along that dependency chain will fix it. Unfortunately, fixing a link is not enough: Every link of the dependency chain between you and the vulnerability needs to be updated before your software will be fixed. Each link must include the fixed version of the thing below it to purge the vulnerability. Thus, the updates need to be done from the bottom up, unless you can eliminate the dependency altogether, which may require similar heroics and is rarely possible—but is the best solution when it is.

Goal: Understand your Options to Remove Vulnerabilities

Today, we lack clarity on this process: what progress has been made by others and what upgrades should be applied at what level? And where is the process stuck? Who is responsible for fixing the vulnerability itself? Who is responsible for propagating the fix?

Goal: Notifications to Speed Repairs

Eventually, your dependencies will be fixed and you can locally upgrade to the new versions. Knowing when this happens is an important goal as it accelerates reducing the exposure to vulnerabilities. We also need a notification system for the actual discovery of vulnerabilities; often new vulnerabilities represent latent problems that are newly discovered even though the actual code has not changed (such as this 10-year old vulnerability in the Unix utility sudo). For large projects, most such issues will arise in the indirect dependencies. Today, we lack the precision required to do notification well, but as we improve vulnerability precision and metadata (as above), we should also drive notification.

So far, we have only described the easy case: a sequence of upgrades that are all backwards compatible, implying that the behavior is the same except for the absence of the vulnerability.

In practice, an upgrade is often not backward compatible, or is blocked by restrictive version requirements. These issues mean that updating a package deep in the dependency tree must cause some churn, or at least requirement updates, in the things above it. The situation often arises when the fix is made to the latest version, say 1.3, but your software or intervening packages request 1.2. We see this situation often, and it remains a big challenge that is made even harder by the difficulty of getting owners to update intervening packages. Moreover, if you use a package in a thousand places, which is not crazy for a big enterprise, you might need to go through the update process a thousand times.

Goal: Fix the Widely Used Versions

It’s also important to fix the vulnerability in the older versions, especially those in heavy use. Such repair is common practice for the subset of software that has long-term support, but ideally all widely used versions should be fixed, especially for security risks.

Automation could help: given a fix for one version, perhaps we can generate good candidate fixes for other versions. This process is sometimes done by hand today, but if we can make it significantly easier, more versions will actually get patched, and there will be less work to do higher in the chain.

To summarize, we need ways to make fixing vulnerabilities, especially in dependencies, both easier and more timely. We need to increase the chance that there is a fix for widely used versions and not just for the latest version, which is often hard to adopt due to the other changes it includes.

Finally, there are many other options on the “fixing” front, including various kinds of mitigations, such as avoiding certain methods, or limiting risk through sandboxes or access controls. These are important practical options that need more discussion and support.

Prevention for Critical Software

The framing above applies broadly to vulnerabilities, regardless of whether they are due to bad actors or are merely innocent mistakes. Although the suggested goals cover most vulnerabilities, they are not sufficient to prevent malicious behavior. To have a meaningful impact on prevention for bad actors, including supply-chain attacks, we need to improve the processes used for development.

This is a big task, and currently unrealistic for the majority of open source. Part of the beauty of open source is its lack of constraints on the process, which encourages a wide range of contributors. However, that flexibility can hinder security considerations. We want contributors, but we cannot expect everyone to be equally focused on security. Instead, we must identify critical packages and protect them. Such critical packages must be held to a range of higher development standards, even though that might add developer friction.

Goal: Define Criteria for “Critical” Open Source Projects that Merit Higher Standards

It is important to identify the “critical” packages that we all depend upon and whose compromise would endanger critical infrastructure or user privacy. These packages need to be held to higher standards, some of which we outline below.

It is not obvious how to define “critical” and the definition will likely expand over time. Beyond obvious software, such as OpenSSL or key cryptographic libraries, there are widely used packages where their sheer reach makes them worth protecting. We started the Criticality Score project to brainstorm this problem with the community, as well collaborating with Harvard on the Open Source Census efforts.

Goal: No Unilateral Changes to Critical Software

One principle that we follow across Google is that changes should not be unilateral—that is, every change involves at least an author and a reviewer or approver. The goal is to limit what an adversary can do on their own—we need to make sure someone is actually looking at the changes. To do this well for open source is actually quite a bit harder than just within a single company, which can have strong authentication and enforce code reviews and other checks.

Avoiding unilateral changes can be broken down into two sub-goals:

Goal: Require Code Review for Critical Software

Besides being a great process for improving code, reviews ensure that at least one person other than the author is looking at every change. Code reviews are a standard practice for all changes within Google.

Goal: Changes to Critical Software Require Approval by Two Independent Parties

To really achieve the “someone is looking” goal, we need the reviewer to be independent from the contributor. And for critical changes, we probably want more than one independent review. We need to sort out what counts as “independent” review, of course, but the idea of independence is fundamental to reviews in most industries.

Goal: Authentication for Participants in Critical Software

Any notion of independence also implies that you know the actors—an anonymous actor cannot be assumed to be independent or trustworthy. Today, we essentially have pseudonyms: the same person uses an identity repeatedly and thus can have a reputation, but we don’t always know the individual’s trustworthiness. This leads to a range of subgoals:

Goal: For Critical Software, Owners and Maintainers Cannot be Anonymous

Attackers like to have anonymity. There have been past supply-chain attacks where attackers capitalized on anonymity and worked their way through package communities to become maintainers, without anyone realizing this “new maintainer” had malicious intent (compromising source code was eventually injected upstream). To mitigate this risk, our view is that owners and maintainers of critical software must not be anonymous.

It is conceivable that contributors, unlike owners and maintainers, could be anonymous, but only if their code has passed multiple reviews by trusted parties.

It is also conceivable that we could have “verified” identities, in which a trusted entity knows the real identity, but for privacy reasons the public does not. This would enable decisions about independence as well as prosecution for illegal behavior.

Goal: Strong Authentication for Contributors of Critical Software

Malicious actors look for easy attack vectors, so phishing attacks and other forms of theft related to credentials are common. One obvious improvement would be the required use of two-factor authentication, especially for owners and maintainers.

Goal: A Federated Model for Identities

To continue the inclusive nature of open source, we need to be able to trust a wide range of identities, but still with verified integrity. This implies a federated model for identities, perhaps similar to how we support federated SSL certificates today—a range of groups can generate valid certificates, but with strong auditing and mutual oversight.

Discussions on this topic are starting to take place in the OpenSSF’s Digital Identity Attestation Working Group.

Goal: Notification for Changes in Risk

We should extend notifications to cover changes in risk. The most obvious is ownership changes, which can be a prelude to new attacks (such as the recent NPM event-stream compromise). Other examples include discovery of stolen credentials, collusion, or other bad actor behavior.

Goal: Transparency for Artifacts

It is common to use secure hashes to detect if an artifact has arrived intact, and digital signatures to prove authenticity. Adding “transparency” means that these attestations are logged publicly and thus document what was intended. In turn, external parties can monitor the logs for fake versions even if users are unaware. Going a step further, when credentials are stolen, we can know what artifacts were signed using those credentials and work to remove them. This kind of transparency, including the durable public logs and the third-party monitoring, has been used to great success for SSL certificates, and we have proposed one way to do this for package managers. Knowing you have the right package or binary is similar to knowing you are visiting the real version of a web site.

Goal: Trust the Build Process

Ken Thompson's Turing Award lecture famously demonstrated in 1984 that authentic source code alone is not enough, and recent events have shown this attack is a real threat. How do you trust your build system? All the components of it must be trusted and verified through a continuous process of building trust.

Reproducible builds help—there is a deterministic outcome for the build and we can thus verify that we got it right—but are harder to achieve due to ephemeral data (such as timestamps) ending up in the release artifact. And safe reproducible builds require verification tools, which in turn must be built verifiably and reproducibly, and so on. We must construct a network of trusted tools and build products.

Trust in both the artifacts and the tools can be established via “delegation”, through a variant of the transparency process described above called binary authorization. Internally, the Google build system signs all artifacts and produces a manifest that ties it to the source code. For open source, one or more trusted agents could run the build as a service, signing the artifact to prove that they are accountable for its integrity. This kind of ecosystem should exist and mostly needs awareness and some agreements on the format of attestations, so that we can automate the processes securely.

The actions in this section are great for software in general, and are essentially in use today within Google, but they are heavier weight than usual for open source. Our hope is that by focusing on the subset of software that is critical, we can achieve these goals at least for that set. As the tooling and automation get better, these goals will become easier to adopt more widely.


The nature of open source requires that we solve problems through consensus and collaboration. For complex topics such as vulnerabilities, this implies focused discussion around the key issues. We presented one way to frame this discussion, and defined a set of goals that we hope will accelerate industry-wide discourse and the ultimate solutions. The first set of goals apply broadly to vulnerabilities and are really about enabling automation and reducing risk and toil.

However, these goals are not enough in the presence of adversaries or to prevent “supply chain” attacks. Thus we propose a second set of goals for critical software. The second set is more onerous and therefore will meet some resistance, but we believe the extra constraints are fundamental for security. The intention is to define collectively the set of “critical” software packages, and apply these higher standards only to this set.

Although we have various opinions on how to meet both sets of goals, we are but one voice in a space where consensus and sustainable solutions matter most of all. We look forward to this discussion, to promoting the best ideas, and eventually to solutions that both strengthen and streamline the security of open source that we all depend on.


  1. Ideally, depended-upon versions should be stable absent an explicit upgrade, but behavior varies depending on the packaging system. Two that aim for stability rather than fast upgrades are Go Modules and NuGet, both of which by default install upgrades only when the requirements are updated; the dependencies might be wrong, but they only change with explicit updates. 

How the Atheris Python Fuzzer Works

On Friday, we announced that we’ve released the Atheris Python fuzzing engine as open source. In this post, we’ll briefly talk about its origins, and then go into lots more detail on how it works.

The Origin Story 

Every year since 2013, Google has held a “Fuzzit”, an internal event where Googlers write fuzzers for their code or open source software. By October 2019, however, we’d already written fuzzers for most of the open-source C/C++ code we use. So for that Fuzzit, the author of this post wrote a Python fuzzing engine based on libFuzzer. Since then, over 50 Python fuzzers have been written at Google, and countless bugs have been reported and fixed.

Originally, this fuzzing engine could only fuzz native extensions, as it did not support Python coverage. But over time, the fuzzer morphed into Atheris, a high-performance fuzzing engine that supports both native and pure-Python fuzzing.

A Bit of Background 

Atheris is a coverage-guided fuzzer. To use Atheris, you specify an “entry point” via atheris.Setup(). Atheris will then rapidly call this entry point with different inputs, hoping to produce a crash. While doing so, Atheris will monitor how the program execution changes based on the input, and will attempt to find new and interesting code paths. This allows Atheris to find unexpected and buggy behavior very effectively.

import atheris

import sys

def TestOneInput(data):  # Our entry point

  if data == b"bad":

    raise RuntimeError("Badness!")


atheris.Setup(sys.argv, TestOneInput)


Atheris is a native Python extension, and uses libFuzzer to provide its code coverage and input generation capabilities. The entry point passed to atheris.Setup() is wrapped in the C++ entry point that’s actually passed to libFuzzer. This wrapper will then be invoked by libFuzzer repeatedly, with its data proxied back to Python.

Python Code Coverage 

Atheris is a native Python extension, and is typically compiled with libFuzzer linked in. When you initialize Atheris, it registers a tracer with CPython to collect information about Python code flow. This tracer can keep track of every line reached and every function executed.

We need to get this trace information to libFuzzer, which is responsible for generating code coverage information. There’s a problem, however: libFuzzer assumes that the amount of code is known at compile-time. The two primary code coverage mechanisms are __sanitizer_cov_pcs_init (which registers a set of program counters that might be visited) and __sanitizer_cov_8bit_counters_init (which registers an array of booleans that are to be incremented when a basic block is visited). Both of these need to know at initialization time how many program counters or basic blocks exist. But in Python, that isn’t possible, since code isn’t loaded until well after Python starts. We can’t even know it when we start the fuzzer: it’s possible to dynamically import code later, or even generate code on the fly.

Thankfully, libFuzzer supports fuzzing shared libraries loaded at runtime. Both __sanitizer_cov_pcs_init and __sanitizer_cov_8bit_counters_init are able to be safely called from a shared library in its constructor (called when the library is loaded). So, Atheris simulates loading shared libraries! When tracing is initialized, Atheris first calls those functions with an array of 8-bit counters and completely made-up program counters. Then, whenever a new Python line is reached, Atheris allocates a PC and 8-bit counter to that line; Atheris will always report that line the same way from then on. Once Atheris runs out of PCs and 8-bit counters, it simply loads a new “shared library” by calling those functions again. Of course, exponential growth is used to ensure that the number of shared libraries doesn’t become excessive.

What's Special about Python 3.8+?

In the README, we advise users to use Python 3.8+ where possible. This is because Python 3.8 added a new feature: opcode tracing. Not only can we monitor when every line is visited and every function is called, but we can actually monitor every operation that Python performs, and what arguments it uses. This allows Atheris to find its way through if statements much better.

When a COMPARE_OP opcode is encountered, indicating a boolean comparison between two values, Atheris inspects the types of the values. If the values are bytes or Unicode, Atheris is able to report the comparison to libFuzzer via __sanitizer_weak_hook_memcmp. For integer comparison, Atheris uses the appropriate function to report integer comparisons, such as __sanitizer_cov_trace_cmp8.

In recent Python versions, a Unicode string is actually represented as an array of 1-byte, 2-byte, or 4-byte characters, based on the size of the largest character in the string. The obvious solution for coverage is to:

  • first compare two strings for equivalent character size and report it as an integer comparison with __sanitizer_cov_trace_cmp8
  • Second, if they’re equal, call __sanitizer_weak_hook_memcmp to report the actual string comparison
However, performance measurements discovered that the surprising best strategy is to convert both strings to utf-8, then compare those with __sanitizer_weak_hook_memcmp. Even with the performance overhead of conversion, libFuzzer makes progress much faster.

Building Atheris

Most of the effort to release Atheris was simply making it build outside of Google’s environment. At Google, building a Python project builds its entire universe of dependencies, including the Python interpreter. This makes it trivial for us to use libFuzzer with our projects - we just compile it into our Python interpreter, along with Address Sanitizer or whatever other features we want.

Unfortunately, outside of Google, it’s not that simple. We had many false starts regarding how to link libFuzzer with Atheris, including making it a standalone shared object, preloading it, etc. We eventually settled on linking it into the Atheris shared object, as it provides the best experience for most users.

However, this strategy still required us to make minor changes to libFuzzer, to allow it to be called as a library. Since most users won’t have the latest Clang and it typically takes several years for distributions to update their Clang installation, actually getting this new libFuzzer version would be quite difficult for most people, making Atheris installation a hassle. To avoid this, we actually patch libFuzzer if it’s too old. Atheris’s will detect an out-of-date libFuzzer, make a copy of it, mark its fuzzer entry point as visible, and inject a small wrapper to allow it to be called via the name LLVMFuzzerRunDriver. If the libFuzzer is sufficiently new, we just call it using LLVMFuzzerRunDriver directly.

The true problem comes from fuzzing native extensions with sanitizers. In theory, fuzzing a native extension with Atheris should be trivial - just build it with -fsanitize=fuzzer-no-link, and make sure Atheris is loaded first. Those magic function calls that Clang injected will point to the libFuzzer symbols inside Atheris. When just fuzzing a native extension without sanitizers, it actually is that simple. Everything works. Unfortunately, sanitizers make everything more complex.

When using a sanitizer like Address Sanitizer with Atheris, it’s necessary to LD_PRELOAD the sanitizer’s shared object. ASan requires that it be loaded first, before anything else; it must either be preloaded, or statically linked into the executable (in this case, the Python interpreter). ASan and UBSan define many of the same code coverage symbols as libFuzzer. In typical libFuzzer usage, this isn’t an issue, since ASan/UBSan declare those symbols weak; the libFuzzer ones take precedence. But when libFuzzer is loaded in a shared object later, that doesn’t work. The symbols from ASan/UBSan have already been loaded via LD_PRELOAD, and coverage information therefore goes to those libraries, leaving libFuzzer very broken.

The only good way to solve this is to link libFuzzer into python itself, instead of Atheris. Since it’s therefore part of the proper executable rather than a shared object that’s dynamically loaded later, symbol resolution works correctly and libFuzzer symbols take precedence. This is nontrivial. We’ve provided documentation about this, and a script to build a modified CPython 3.8.6. These scripts will use the same possibly-patched libFuzzer as Atheris.

Why is it called Atheris? 

Atheris Hispida, or the “Hairy bush viper”, is the closest thing that exists to a fuzzy Python.

OpenTitan at One Year: the Open Source Journey to Secure Silicon

During the past year, OpenTitan has grown tremendously as an open source project and is on track to provide transparent, trustworthy, and cost-free security to the broader silicon ecosystem. OpenTitan, the industry’s first open source silicon root of trust, has rapidly increased engineering contributions, added critical new partners, selected our first tapeout target, and published a comprehensive logical security model for the OpenTitan silicon, among other accomplishments.

OpenTitan by the Numbers 

OpenTitan has doubled many metrics in the year since our public launch: in design size, verification testing, software test suites, documentation, and unique collaborators at least. Crucially, this growth has been both in the design verification collateral required for high volume production-quality silicon, as well as the digital design itself, a first for any open source silicon project.
  • More than doubled the number of commits at launch: from 2,500 to over 6,100 (across OpenTitan and the Ibex RISC-V core sub-project).
  • Grew to over 141K lines of code (LOC) of System Verilog digital design and verification.
  • Added 13 new IP blocks to grow to a total to 29 distinct hardware units.
  • Implemented 14 Device Interface Functions (DIFs) for a total 15 KLOC of C11 source code and 8 KLOC of test software.
  • Increased our design verification suite to over 66,000 lines of test code for all IP blocks.
  • Expanded documentation to over 35,000 lines of Markdown.
  • Accepted contributions from 52 new unique contributors, bringing our total to 100.
  • Increased community presence as shown by an aggregate of over 1,200 Github stars between OpenTitan and Ibex.

One year of OpenTitan and Ibex growth on GitHub: the total number of commits grew from 2,500 to over 6,100.

High quality development is one of OpenTitan’s core principles. Besides our many style guides, we require thorough documentation and design verification for each IP block. Each piece of hardware starts with auto-generated documentation to ensure consistency between documentation and design, along with extensive, progressively improving, design verification as it advances through the OpenTitan hardware stages to reach tapeout readiness.

One year of growth in Design Verification: from 30,000 to over 65,000 lines of testing source code. Each color represents design verification for an individual IP block.

Innovating for Open Silicon Development

Besides writing code, we have made significant advances in developing processes and security framework for high quality, secure open source silicon development. Design success is not just measured by the hardware, highly functional software and a firm contract between the two, with well-defined interfaces and well-understood behavior, play an important role.

OpenTitan’s hardware-software contract is realized by our DIF methodology, yet another way in which we ensure hardware IP quality. DIFs are a form of hardware-software co-design and the basis of our chip-level design verification testing infrastructure. Each OpenTitan IP block requires a style guide-compliant DIF, and this year we implemented 14 DIFs for a total 15 KLOC of C11 source code and 8 KLOC of tests.

We also reached a major milestone by publishing an open Security Model for a silicon root of trust, an industry first. This comprehensive guidance demonstrates how OpenTitan provides the core security properties required of a secure root of trust. It covers provisioning, secure boot, device identity, and attestation, and our ownership transfer mechanism, among other topics.

Expanding the OpenTitan Ecosystem 

Besides engineering effort and methodology development, the OpenTitan coalition added two new Steering Committee members in support of lowRISC as an open source not-for-profit organization. Seagate, a leader in storage technology, and Giesecke and Devrient Mobile Security, a major producer of certified secure systems. We also chartered our Technical Committee to steer technical development of the project. Technical Committee members are drawn from across our organizational and individual contributors, approving 9 technical RFCs and adding 11 new project committers this past year. 

On the strength of the OpenTitan open source project’s engineering progress, we are excited to announce today that Nuvoton and Google are collaborating on the first discrete OpenTitan silicon product. Much like the Linux kernel is itself not a complete operating system, OpenTitan’s open source design must be instantiated in a larger, complete piece of silicon. We look forward to sharing more on the industry’s first open source root of trust silicon tapeout in the coming months.

Onward to 2021

OpenTitan’s future is bright, and as a project it fully demonstrates the potential for open source design to enable collaboration across disparate, geographically far flung teams and organizations, to enhance security through transparency, and enable innovation in the open. We could not do this without our committed project partners and supporters, to whom we owe all this progress: Giesecke and Devrient Mobile Security, Western Digital, Seagate, the lowRISC CIC, Nuvoton, ETH Zürich, and many independent contributors.

Interested in contributing to the industry's first open source silicon root of trust? Contact us here.

Improving open source security during the Google summer internship program

Every summer, Google’s Information Security Engineering (ISE) team hosts a number of interns who work on impactful projects to help improve security at Google. This year was no different—well, actually it was a little bit different because internships went virtual. But our dedication to security was still front and center as our intern team worked on improvements in open source software.

Open source software is the foundation of many modern software products. Over the years, developers increasingly have relied on reusable open source components for their applications. It is paramount that these open source components are secure and reliable. 

The focus of this year’s intern projects reflects ISE’s general approach of tackling security issues at scale, and can be split into three main areas: 
  • Vulnerability research: Finding new vulnerabilities, developing infrastructure to search for known bug classes at scale, and experimenting with new detection approaches.
  • Mitigation and hardening: Developing hardening approaches with the goal of fully eliminating specific vulnerability classes or mitigating their impact.
  • Security education: Sharing knowledge to increase awareness among developers and to help train security engineers.
Vulnerability research

Fuzzing is a highly effective method of uncovering memory-corruption vulnerabilities in C and C++ applications. With OSS-Fuzz, Google provides a platform for fuzzing open source software. One of this year’s intern projects ported internal fuzz targets to OSS-Fuzz, which led to the discovery of new bugs. In this context, our interns experimented with setting up fuzzing for difficult fuzz targets such as the state machines of Memcached and Redis. Additionally, they added new fuzzers for complicated targets like nginx, PostgreSQL, and Envoy, a widely used cloud-native high-performance proxy. 

State-of-the-art fuzzing frameworks like AFL, libFuzzer, and Honggfuzz leverage feedback such as code coverage to guide the fuzzer. Recent academic papers suggest that symbolic execution can complement existing fuzzing frameworks to find bugs that are difficult for random mutation-based fuzzers to find. Our interns evaluated the possibility of using KLEE to augment libFuzzer and AFL. In particular, they found that adding KLEE to existing fuzzing frameworks provides benefits for fuzz targets such as sqlite and lcms. However, at this point in time, there is still work to be done before symbolic execution can be performed at scale (e.g., in OSS-Fuzz).

In addition to finding memory-corruption vulnerabilities, fuzzing can help find logic vulnerabilities. This can be difficult as it requires understanding the semantics of the target application. One approach uses differential testing to find different behaviors in applications that are supposed to behave in the same way. One of our intern projects this summer looked into leveraging differential fuzzing to expose logic vulnerabilities and found a number of cases where YAML parsers handle edge cases differently.

Other intern projects this summer focused on the search for application-specific vulnerabilities. Our interns aimed to discover common Google Kubernetes Engine (GKE) misconfigurations. The recently launched GKE-Auditor, created by one of our interns, implements 18 detectors to find misconfigurations in Node isolation, role-based access control, and pod security policies. Another project implemented regression tests for the Google Compute Engine (GCE) metadata server

Finally, one intern project looked into improving Visual Studio Code (VSCode), a popular cross-platform code editor that is based on Electron which combines the Chromium rendering engine and the Node.js runtime. VSCode can be vulnerable to DOM cross-site scripting attacks. For this reason, our intern’s work centered on making VSCode Trusted Types-compliant by using and contributing to the static and dynamic analysis tools to find violations. This work not only led to an improvement of VSCode, but also of Chromium.


Because finding all vulnerabilities is an impossible task, we always look for ways to mitigate their impact or eliminate certain vulnerability classes completely. The main focus of this year’s hardening projects were to enable security enhancements for major web frameworks and to provide sandboxing for popular libraries written in memory-unsafe languages such as C and C++.

In an effort to make the web more secure, our intern team added security enhancements including Content Security Policy (CSP), Fetch Metadata Request Headers, Cross-Origin Opener Policy (COOP), and Cross-Origin Embedder Policy (COEP) to a number of existing web frameworks (our previous post provides a good overview of these mitigations).

As a result, these web security features were implemented in a number of common application frameworks, including Apache Struts [CSP, COOP/COEP], Apache Wicket [Fetch Metadata, COOP/COEP], .NET Core [CSP], Django [Trusted Types, COOP], and WordPress [Fetch Metadata, CSP]. We're looking forward to working with open source maintainers to further develop and integrate these defenses into more popular frameworks!


Executing native code that comes from untrusted origins or processes data from untrusted sources is risky because it may be malicious or contain vulnerabilities. Sandboxing mitigates these risks by executing code in a low-privileged environment.This process often requires modifying the interfaces of third-party libraries and setting up their execution environment. Sandboxed API is a framework to help with these tasks that is used at Google. 

Our interns also worked on providing reusable sandboxes for popular open source libraries such as curl, OpenJPEG, LoadPNG, LibUV, and libTIFF. Now, anyone who wants to use these libraries to process untrusted data can do so safely.


Capture the flag (CTF) competitions are useful for transferring security knowledge and training security engineers. The kCTF project provides a Kubernetes-based infrastructure which offers a hardened environment to securely deploy CTF tasks and isolate them from each other. One intern project added a number of improvements to the documentation including enabling a version control to allow multiple authors to work on one challenge and simplifingkCTF’s usage.

We would like to thank all of our interns for their hard work this summer! For more information on the Google internship program and other student opportunities, check out

Fostering research on new web security threats

The web is an ecosystem built on openness and composability. It is an excellent platform for building capable applications, and it powers thousands of services created and maintained by engineers at Google that are depended on by billions of users. However, the web's open design also allows unrelated applications to sometimes interact with each other in ways which may undermine the platform's security guarantees.

Increasingly, security issues discovered in modern web applications hinge upon the misuse of long-standing web platform behaviors, allowing unsavory sites to reveal information about the user or their data in other web applications. This class of issues, broadly referred to as cross-site leaks (XS-Leaks), poses interesting challenges for security engineers and web browser developers due to a diversity of attacks and the complexity of building comprehensive defenses.

To promote a better understanding of these issues and protect the web from them, today marks the launch of the XS-Leaks wiki—an open knowledge base to which the security community is invited to participate, and where researchers can share information about new attacks and defenses.

The XS-Leaks wiki 

Available at (code on GitHub), the wiki explains the principles behind cross-site leaks, discusses common attacks, and proposes defense mechanisms aimed at mitigating these attacks. The wiki is composed of smaller articles that showcase the details of each cross-site leak, their implications, proof-of-concept code to help demonstrate the issue, and effective defenses. 

To improve the state of web security, we're inviting the security community to work with us on expanding the XS-Leaks wiki with information about new offensive and defensive techniques.


An important goal of the wiki is to help web developers understand the defense mechanisms offered by web browsers that can comprehensively protect their web applications from various kinds of cross-site leaks. 

Each attack described in the wiki is accompanied by an overview of security features which can thwart or mitigate it; the wiki aims to provide actionable guidance to assist developers in the adoption of new browser security features such as Fetch Metadata Request Headers, Cross-Origin Opener Policy, Cross-Origin Resource Policy, and SameSite cookies.

The Security Team at Google has benefited from over a decade of productive collaboration with security experts and browser engineers to improve the security of the web platform. We hope this new resource encourages further research into creative attacks and robust defenses for a major class of web security threats. We're excited to work together with the community to continue making the web safer for all users.

Special thanks to Manuel Sousa for starting the wiki as part of his internship project at Google, and to the contributors to the xsleaks GitHub repository for their original research in this area.

Announcing our open source security key test suite

Security keys and your phone’s built-in security keys are reshaping the way users authenticate online. These technologies are trusted by a growing number of websites to provide phishing-resistant two-factor authentication (2FA). To help make sure that next generation authentication protocols work seamlessly across the internet, we are committed to partnering with the ecosystem and providing essential technologies to advance state-of-the-art authentication for everyone. So, today we are releasing a new open source security key test suite

The protocol powering security keys

Under the hood, roaming security keys are powered by the FIDO Alliance CTAP protocols, the part of FIDO2 that ensures a seamless integration between your browser and security key. Whereas the security-key user experience aims to be straightforward, the CTAP protocols themselves are fairly complex. This is due to the broad range of authentication use cases the specification addresses: including websites, operating systems, and enterprise credentials. As the protocol specification continues to evolve—there is already a draft of CTAP 2.1—corner cases that can cause interoperability problems are bound to appear.

Building a test suite  

We encountered many of those tricky corner cases while implementing our open-source security-key firmware OpenSK and decided to create a comprehensive test suite to ensure all our new firmware releases handle them correctly. Over the last two years, our test suite grew to include over 80 tests that cover all the CTAP2 features.

Strengthening the ecosystem 

A major strength of the security key ecosystem is that the FIDO Alliance is an industry consortium with many participating vendors providing a wide range of distinct security keys catering to all users' needs. The FIDO Alliance offers testing for conformance to the current specifications. Those tests are a prerequisite to passing the interoperability tests that are required for a security key to become FIDO Certified. Our test suite complements those official tools by covering additional scenarios and in-market corner cases that are outside the scope of the FIDO Alliance’s testing program.

Back in March 2020, we demonstrated our test suite to the FIDO Alliance members and offered to extend testing to all FIDO2 keys. We got an overwhelmingly positive response from the members and have been working with many security key vendors since then to help them make the best use of our test suite.

Overall, the initial round of the tests on several keys has yielded promising results and we are actively collaborating with many vendors on building on those results to improve future keys.

Open-sourcing our test suite 

Today we are making our test suite open source to allow security key vendors to directly integrate it into their testing infrastructure and benefit from increased testing coverage. Moving forward, we are excited to keep collaborating with the FIDO Alliance, its members, the hardware security key industry and the open source community to extend our test suite to improve its coverage and make it a comprehensive tool that the community can rely on to ensure key interoperability. In the long term, it is our hope that strengthening the community testing capabilities will ultimately benefit all security key users by helping ensure they have a consistent experience no matter which security keys they are using.


We thank our collaborators: Adam Langley, Alexei Czeskis, Arnar Birgisson, Borbala Benko, Christiaan Brand, Dirk Balfanz, Guillaume Endignoux, Jeff Hodges, Julien Cretin, Mark Risher, Oxana Comanescu, Tadek Pietraszek and all the security key vendors that worked with us.

Making the Advanced Protection Program and Titan Security Keys easier to use on Apple iOS devices

Starting today, we’re rolling out a change that enables native support for the W3C WebAuthn implementation for Google Accounts on Apple devices running iOS 13.3 and above. This capability, available for both personal and work Google Accounts, simplifies your security key experience on compatible iOS devices and allows you to use more types of security keys for your Google Account and the Advanced Protection Program.

Using an NFC security key on iPhone

More security key choices for users
  • Both the USB-A and Bluetooth Titan Security Keys have NFC functionality built-in. This allows you to tap your key to the back of your iPhone when prompted at sign-in.
  • You can use a Lightning security key like the YubiKey 5Ci or any USB security key if you have an Apple Lightning to USB Camera Adapter.
  • You can plug a USB-C security key in directly to an iOS device that has a USB-C port (such as an iPad Pro).
  • We suggest installing the Smart Lock app in order to use Bluetooth security keys and your phone’s built-in security key, which allows you to use your iPhone as an additional security key for your Google Account.
In order to add your Google Account to your iOS device, navigate to “Settings > Passwords & Accounts” on your iOS device or install the Google app and sign in.

Account security best practices

We highly recommend users at a higher risk of targeted attacks to get security keys (such as Titan Security Key or your Android or iOS phone) and enroll into the Advanced Protection Program. If you’re working for political committees in the United States, you may be eligible to request free Titan Security Keys through the Defending Digital Campaigns to get help enrolling into Advanced Protection.

You can also use security keys for any site where FIDO security keys are supported for 2FA, including your personal or work Google Account, 1Password, Bitbucket, Bitfinex, Coinbase, Dropbox, Facebook, GitHub, Salesforce, Stripe, Twitter, and more.