Author Archives: Kaylin Trychon

Find and $eek! Increased rewards for Google Nest & Fitbit devices

Posted by Medha Jain, Program Manager, Devices & Services Security 

At Google, we constantly invest in security research to raise the bar for our devices, keeping our users safe and building their trust in our products. In 2021, we published Google Nest security commitments, in which we committed to engage with the research community to examine our products and services and report vulnerabilities.

We are now looking to deepen this relationship and accelerate the path toward building more secure devices. Starting today, we will introduce a new vulnerability rewards structure for submissions impacting smart home (Google Nest) and wearables (Fitbit) devices through our Bug Hunters platform.

Bonus!

We are paying higher rewards retroactively for eligible Google Nest and Fitbit devices reports submitted in 2021. And, starting today, for the next six months, will double the reward amount for all new eligible reports applicable to Google Nest & Fitbit devices in scope.

We will continue to take reports on our web applications, services, and mobile apps at their existing reward levels. Please keep those coming!

An enhanced rewards program

Building on our previous programs to improve devices' embedded security posture, we’re bringing all our first-party devices under a single program, starting with Google Nest, Fitbit, and Pixel.
This program extends the Android Security Reward Program, making it easier for researchers to submit a vulnerability in first-party devices and improving consistency across our severity assignments. Refer to the Android and Google Devices Security Reward Program for more details.

What interests us?

We encourage researchers to report firmware, system software, and hardware vulnerabilities. Our wide diversity of platforms provides researchers with a smorgasbord of environments to explore.

What's next?

We will be at the Hardwear.io conference this year! The VRP team is looking forward to meeting our security peers in person. We’ll be talking about the architecture of a couple of our devices, hoping to give security researchers a head start in finding vulnerabilities. We’ll have plenty of swag, too!

We will continue to enhance the researchers' experience and participation. We intend to add training documentations and target areas that interest us as we grow the program.

A huge thanks to Sarah Jacobus, Adam Bacchus, Ankur Chakraborty, Eduardo' Vela" <Nava>, Jay Cox, and Nic Watson.

Mitigating kernel risks on 32-bit ARM

Posted by Ard Biesheuvel, Google Open Source Security Team 

Linux kernel support for the 32-bit ARM architecture was contributed in the late 90s, when there was little corporate involvement in Linux development, and most contributors were students or hobbyists, tinkering with development boards, often without much in the way of documentation.



Now 20+ years later, 32-bit ARM's maintainer has downgraded its support level to 'odd fixes,' while remaining active as a kernel contributor. This is a common pattern for aging and obsolete architectures: corporate funding for Linux kernel development has tremendously increased the pace of development, but only for architectures with a high return on investment. As a result, the 32-bit ARM port of Linux is essentially in maintenance-only mode, and lacks core Linux advancements such as THREAD_INFO_IN_TASK or VMAP_STACK, which protect against stack overflow attacks.

The lack of developer attention does not imply that the 32-bit ARM port has ceased to make economic sense, though. Instead, it has evolved from being one of the spearheads of Linux innovation to a stable and mature platform, and while funding its upstream development may not make sense in the long term, deploying 32-bit ARM into the field today most certainly still makes economic sense when margins are razor thin and BOM costs need to be kept to an absolute minimum. This is why 32-bit ARM is still widely used in embedded systems like set-top boxes and wireless routers.

Running 32-bit Linux on 64-bit ARM systems

Ironically, at these low price points, the DRAM is actually the dominant component in terms of BOM cost, and many of these 32-bit ARM systems incorporate a cheap ARMv8 SoC that happens to be capable of running in 64-bit mode as well. The reason for running 32-bit applications nonetheless is that these generally use less of the expensive DRAM, and can be deployed directly without the need to recompile the binaries. As 32-bit applications don't need a 64-bit kernel (which itself uses more memory due to its internal use of 64-bit pointers), the product ships with a 32-bit kernel instead.


If you're choosing to use a 32-bit kernel for its smaller memory footprint, it's not without risks. You'll likely experience performance issues, unpatched vulnerabilities, and unexpected misbehaviors such as:

  • 32-bit kernels generally cannot manage more than 1 GiB of physical memory without resorting to HIGHMEM bouncing, and cannot provide a full virtual address space of 4 GiB to user space, as 64-bit kernels can.
  • Side channels or other flaws caused by silicon errata may exist that haven't been mitigated in 32-bit kernels. For example, the hardening against Spectre and Meltdown vulnerabilities were only done for ARMv7 32-bit only CPUs, and many ARMv8 cores running in 32-bit mode may still be vulnerable (only Cortex-A73 and A75 are handled specifically). And in general, silicon flaws in 64-bit parts that affect the 32-bit kernel are less likely to be found or documented, simply because the silicon validation teams don’t prioritize them.
  • The 32-bit ARM kernel does not implement the elaborate alternatives patching framework that is used by other architectures to implement handling of silicon errata, which are particular to certain revisions of certain CPUs. Instead, on 32-bit multiplatform kernels, we simply enable all errata workarounds that may be needed by any of the cores that may ever run the image in question, potentially affecting performance unnecessarily on cores that have no need for them.
  • Silicon vendors are phasing out 32-bit support in the longer term. Given an ecosystem containing a handful of operating systems and thousands of applications, support for 32-bit operating systems (which is more complex technically) is highly likely to be dropped first. For products with longer life cycles, long-term procurement contracts for components available today are usually much more costly than adjusting the BOM over time and using newer, cheaper parts.
  • The 32-bit kernel does not implement kernel address space randomization, and even if it did, its comparatively tiny address space simply leaves very little space for randomization. Other hardening features, such as rodata=full or hierarchical eXecute Never attributes, are missing as well on 32-bit, and are not likely to be implemented, either due to lack of support in the architecture, or because of the complexity of the 32-bit memory management code, which still supports all of the different architecture revisions dating back to the initial Linux port running on the Risc PC.
Keeping the 32-bit ARM kernel secure

There are cases, though, where using the 32-bit kernel is the only option, e.g., if the CPUs are in fact 32-bit only (which is the case even for some ARMv8 cores such as Cortex-A32), or when relying on an existing 32-bit only codebase running in the kernel (drivers for legacy peripherals). Note that in such cases, it still makes sense to use the most recent kernel version compatible with the hardware, since we are in fact making an effort to enable some of the existing hardening features on 32-bit ARM as well.

  • THREAD_INFO_IN_TASK for v7 SMP cores
The v5.16 release of the Linux kernel implements support for THREAD_INFO_IN_TASK when running on ARMv7 SMP systems. This protects the kernel's per-task bookkeeping (called thread_info), which lives on the far (and normally unused) end of the stack, against stack overflows which may occur in rare -yet sometimes exploitable- cases where the control flow of the program simply ends up accumulating more state than the stack can hold. (Note that a stack overflow is not the same as a stack buffer overflow, where the overflow happens in the opposite direction.)

By moving thread_info off the stack and into the kernel heap, and by using a special SMP CPU register to keep track of its location, we can mitigate the risk of stack overflows resulting in thread_info corruption. However, it does not prevent stack overflows themselves: these may still occur, and result in corruption of other data structures that happen to be adjacent to the task stack in memory.
  • THREAD_INFO_IN_TASK for other cores
For CPUs that lack this special SMP CPU register, we also proposed an implementation of THREAD_INFO_IN_TASK that is expected to land in v5.18. Instead of a special register, it uses a global variable to keep track of the location of thread_info.


  • VMAP_STACK support
Preventing stack overflows from corrupting unrelated memory contents is the goal of VMAP_STACK, which we are enabling for 32-bit ARM as well. When VMAP_STACK is enabled, kernel mode stacks are allocated from the kernel heap as before, but mapped into a different part of the kernel's address space, and surrounded by guard regions, which are guaranteed to be kept unpopulated. Given that accesses to such unpopulated regions will trigger an exception, the kernel's memory management layer can step in and terminate the program as soon as a stack overflow occurs, and prevent it from causing memory corruption.


Support for IRQ stacks

Coming up with a bounded worst case on which to base the size of the kernel stack is rather hard, especially given the fact that it is shared between the program itself and any exception handling routines that may be called on its behalf, including interrupt handlers. To mitigate the risk of a pathological worst case occurring, where an interrupt fires that needs a lot of stack space right at a time when most of the stack is already being used by the program, we are also enabling IRQ_STACKS for 32-bit ARM, which will run handlers of both hard and soft interrupts from a dedicated stack, one for each CPU. By decoupling the task and interrupt contexts like this, the likelihood that a well-behaved program needs to be terminated due to stack overflow should be all but eliminated.

Conclusion

With these changes in place, kernel stack overflow protection will be available for all ARM systems supported by Linux, including ancient ones like the Risc PC or Netwinder, provided that it runs a Linux distribution that is keeping up with the times.




However, relying on legacy hardware and software comes with a risk, and even though we try to help keep users of the 32-bit kernel as safe as we reasonably can, it is not the right choice for new designs that incorporate 64-bit capable hardware.

? Roses are red, Violets are blue ? Giving leets ?‍? more sweets ? All of 2022!

Posted by Eduardo Vela, Vulnerability Matchmaker 

Until December 31 2022 we will pay 20,000 to 91,337 USD for exploits of vulnerabilities in the Linux Kernel, Kubernetes, GKE or kCTF that are exploitable on our test lab.

We launched an expansion of kCTF VRP on November 1, 2021 in which we paid 31,337 to 50,337 USD to those that are able to compromise our kCTF cluster and obtain a flag. We increased our rewards because we recognized that in order to attract the attention of the community we needed to match our rewards to their expectations. We consider the expansion to have been a success, and because of that we would like to extend it even further to at least until the end of the year (2022).

During the last three months, we received 9 submissions and paid over 175,000 USD so far. The submissions included five 0days and two 1days. Three of these are already fixed and are public: CVE-2021-4154, CVE-2021-22600 (patch) and CVE-2022-0185 (writeup). These three bugs were first found by Syzkaller, and two of them had already been fixed on the mainline and stable versions of the Linux Kernel at the time they were reported to us.

Based on our experience these last 3 months, we made a few improvements to the submission process:
  • Reporting a 0day will not require including a flag at first. We heard some concerns from participants that exploiting a 0day in the shared cluster could leak it to other participants. As such, we will only ask for the exploit checksum (but you still have to exploit the bug and submit the flag within a week after the patch is merged on mainline). Please make sure that your exploit works on COS with minimal modifications (test it on your own kCTF cluster), as some common exploit primitives (like eBPF and userfaultfd) might not be available.
  • Reporting a 1day will require including a link to the patch. We will automatically publish the patches of all submissions if the flag is valid. We also encourage you all to include a link to a Syzkaller dashboard report if applicable in order to help reduce duplicate submissions and so you can see which bugs were exploited already.
  • You will be able to submit the exploit in the same form you submit the flag. If you had submitted an exploit checksum for a 0day, please make sure that you include the original exploit as well as the final exploit and make sure to submit it within a week after the patch is merged on mainline. The original exploit shouldn't require major modifications to work. Note that we need to be able to understand your exploit, so please add comments to explain what it is doing.
  • We are now running two clusters, one on the REGULAR release channel and another one on the RAPID release channel. This should provide more flexibility whenever a vulnerability is only exploitable on modern versions of the Linux Kernel or Kubernetes.
We are also changing the reward structure slightly. Going forward the rewards will be:
  • 31,337 USD to the first valid exploit submission for a given vulnerability. This will only be paid once per vulnerability and only once per cluster version/build (available at /etc/node-os-release).
  • 0 USD for exploits for duplicate exploits for the same vulnerability. The bonuses below might still apply.
Bonuses
  • 20,000 USD for exploits for 0day vulnerabilities. This will only be paid once per vulnerability to the first valid exploit submission.
    • To submit 0days, please test your exploit (we recommend to test it on your own kCTF cluster to avoid leaking it to other participants), make a checksum and send the checksum to us. Within a week after the vulnerability is fixed on the mainline, submit the form as a 1day and include the exploit of which you sent a checksum to us.
  • 20,000 USD for exploits for vulnerabilities that do not require unprivileged user namespaces (CLONE_NEWUSER). This will only be paid once per vulnerability to the first valid exploit submission.
    • Our test lab allows unprivileged user namespaces, so we will manually check the exploits to check if they work without unprivileged user namespaces when deciding whether to issue the bonus. We decided to issue additional rewards for exploits that do not require unprivileged user namespaces because containers default seccomp policy does not allow the use of unprivileged user namespaces on containers that are run without CAP_SYS_ADMIN. This feature is now available on Kubernetes and all nodes running on GKE Autopilot have it enabled by default.
  • 20,000 USD for exploits using novel exploit techniques. This is a bonus in addition to the base rewards (applies for duplicate exploits). To qualify for this additional reward please send us a write-up explaining it.
    • An example of something considered as a novel technique could be the exploitation of previously unknown objects to transform a limited primitive into a more powerful one, such as an arbitrary/out-of-bounds read/write or arbitrary free. For example, in all our submissions, researchers leveraged message queues to achieve kernel information leaks. We are looking for similarly powerful techniques that allow heap exploits to be “plugged in” and immediately allow kernel access. Another example is bypassing a common security mitigation or a technique for exploiting a class of vulnerabilities more reliably.
These changes increase some 1day exploits to 71,337 USD (up from 31,337 USD), and makes it so that the maximum reward for a single exploit is 91,337 USD (up from 50,337 USD). We also are going to pay even for duplicates at least 20,000 USD if they demonstrate novel exploit techniques (up from 0 USD). However, we will also limit the number of rewards for 1days to only one per version/build. There are 12-18 GKE releases per year on each channel, and we have two clusters on different channels, so we will pay the 31,337 USD base rewards up to 36 times (no limit for the bonuses). While we don't expect every upgrade to have a valid 1day submission, we would love to learn otherwise. You can find the flag submission status for our clusters (and their versions) here.

We look forward to hearing from you, and continue to strengthen our shared ecosystem. If you are interested to participate but don't know where to start, Arizona State University has a free public Kernel Exploitation workshop at https://dojo.pwn.college/challenges/kernel as part of an overall memory corruption course and you can find a community-maintained list of past Linux Kernel vulnerabilities, exploits and writeups curated by Andrey Konovalov at https://github.com/xairy/linux-kernel-exploitation.

This is part of our Vulnerability Reward Program, which we've been running for over 10 years, and the rules include some more information. Same as with our other rewards, we will double them if they are donated to charity, and submitters will be included on our site at bughunters.google.com. If you are ready to submit something, please read the instructions on our site here and if you have any other questions please contact us on Discord.

Vulnerability Reward Program: 2021 Year in Review

Posted by Sarah Jacobus, Vulnerability Rewards Team 

Last year was another record setter for our Vulnerability Reward Programs (VRPs). Throughout 2021, we partnered with the security researcher community to identify and fix thousands of  vulnerabilities – helping keep our users and the internet safe. 


Thanks to these incredible researchers, Vulnerability Reward Programs across Google continued to grow, and we are excited to report that in 2021 we awarded a record breaking $8,700,000 in vulnerability rewards – with researchers donating over $300,000 of their rewards to a charity of their choice. 


We also launched bughunters.google.com in 2021, a public researcher portal dedicated to keeping Google products and the internet safe and secure. This new platform brings all of our VRPs (Google, Android, Abuse, Chrome, and Google Play) closer together and provides a single intake form, making security bug submission easier than ever. We’re excited about everything the new Bug Hunters portal has to offer, including:


  • More opportunities for interaction and a bit of healthy competition through gamification, per-country leaderboards, awards/badges for certain bugs, and more!

  • A more functional and aesthetically pleasing leaderboard. We know a lot of you are using your achievements in our VRPs to find jobs (we’re hiring!) and we hope this acts as a useful resource.

  • A stronger emphasis on learning: bug hunters can improve their skills through the content available in our new Bug Hunter University

  • Streamlined publication process: we know the value that knowledge sharing brings to our community. That’s why we want to make it easier for you to publish your bug reports.

  • We now offer swag! The first 20 folks who share this blog post on Twitter and tag @GoogleVRP will receive a gift voucher for swag in their DMs.  


As in past years, we are sharing our 2021 Year in Review statistics across all of our programs. We would like to give a special thank you to all of our dedicated researchers - we look forward to more collaboration in the future!




Android


The Android VRP doubled its 2020 total payouts in 2021 with nearly $3 million dollars in rewards, and awarded the highest payout in Android VRP history: an exploit chain discovered in Android receiving a reward of $157,000!


Our industry leading prize of $1,500,000 for a compromise of our Titan-M Security chip used in our Pixel device remains unclaimed - for more information on this reward and Android exploit chain rewards, please visit our public rules page. 


The program also launched the Android Chipset Security Reward Program (ACSRP), a vulnerability reward program offered by Google in collaboration with manufacturers of certain popular Android chipsets. This private, invite-only program, provides reward and recognition for contributions of security researchers who invest their time and effort into helping make Android devices more secure. In 2021 the ACSRP paid out $296,000 for over 220 valid and unique security reports.  


We would like to give a special shoutout to some of our top researchers whose continued hard work keeps Android safe and secure:


  • Aman Pandey of Bugsmirror Team has skyrocketed to our top researcher last year, submitting 232 vulnerabilities in 2021! Since submitting their first report in 2019, Aman has reported over 280 valid vulnerabilities to the Android VRP and has been a crucial part of making our program so successful.

  • Yu-Cheng Lin (林禹成) (@AndroBugs) has been another phenomenal researcher for the Android VRP, submitting a whopping 128 valid reports to the program in 2021. 

  • Researcher [email protected] discovered a critical exploit chain in Android (CVE-2021-39698) , receiving the highest payout in Android VRP history of $157,000. 


Chrome

This year the Chrome VRP also set some new records – 115 Chrome VRP researchers were rewarded for 333 unique Chrome security bug reports submitted in 2021, totaling $3.3 million in VRP rewards. The contributions not only help us to improve Chrome, but also the web at large by bolstering the security of all browsers based on Chromium.


Of the $3.3 million, $3.1 million was awarded for Chrome Browser security bugs and $250,500 for Chrome OS bugs, including a $45,000 top reward amount for an individual Chrome OS security bug report and $27,000 for an individual Chrome Browser security bug report.



Of these totals, $58,000 was awarded for security issues discovered by fuzzers contributed by VRP researchers to the Chrome Fuzzing program. Each valid report from an externally provided fuzzer received a $1,000 patch bonus, with one fuzzer report receiving a $16,000 reward.


The Chrome VRP would not be able to smash these records over the last year without the efforts of so many exceptional VRP researchers. We’d like to highlight a few researcher achievements made in 2021:


  • Rory McNamara, a Chrome OS VRP researcher who has been participating in the Chrome VRP for five years, became the highest awarded Chrome VRP researcher of all time. This year he was rewarded for six reports achieving root privilege escalation in Chrome OS, one of which received the highest reward amount achieved for a single Chrome bug report in 2021 at $45,000. 


  • Chrome Browser VRP researcher Leecraso (@leecraso) of 360 Vulnerability Research Institute was the most awarded researcher of 2021, with 18 valid bug reports; a majority of which were for memory corruption vulnerabilities affecting the browser process.

 

  • We love when researchers write about their findings (only after we have publicly disclosed the bug, of course)! Chrome Browser VRP researcher Brendon Tiszka wrote an excellent two-part blog series on his discovery and exploitation of a V8 vulnerability, CVE-2021-21225, the analysis and reporting of which earned him a $22,000 VRP reward.


Huge thanks and congratulations to all Chrome VRP researchers that helped us make Chrome and Chrome OS more safe for all users in 2021!.



Google Play

Google Play paid out $550,000 in rewards to over 60 unique security researchers.


The Google Play Security Reward Program also released their Android App Hacking Workshop content and published a blog on their work to empower the next generation of Android Application Security Researchers. 


kCTF VRP


In November we expanded our reward amounts for exploits against our kCTF cluster from 5,000-10,000 up to 31,337-50,337 USD. In the last 3 months we were happy to have several participants receive $175,685 USD in rewards. We also extended the timeline of the increased rewards until February 14 (from January 31) which should give everyone a couple more weeks to finalize any almost-working exploits.


GCP VRP Prize


To encourage security researchers to focus on Google Cloud Platform, we initiated the annual GCP VRP Prize in 2019. In March this year, we announced the winners of the 2020 edition of the prize and paid out $313,337 in prizes. Ezequiel Pereira won the top prize of $133,337 for finding an RCE in Google Cloud Deployment Manager. We saw some amazing research on Google Cloud Platform this year too. Stay tuned for the 2021 winners!


Research Grants

Six years ago, the Google VRP launched an experimental Vulnerability Research Grant program to encourage seasoned security researchers to take a detailed and extensive look into the security of Google products and services. And reward them even if there are no vulnerabilities found. Six years later, we are happy to announce that in 2021 we awarded over $200,000 in grants to more than 120 security researchers around the world. 


If you are a Google VRP researcher and want to be considered for a Vulnerability Research Grant make sure you opted in on your bughunters profile.


Looking forward


With the launch of the new Bug Hunters portal, we plan to continue improving our platform and listening to you - our researchers - on ways we can improve our platform and Bug Hunter University. 


Thank you again for making Google, the Internet, and our users safe and secure! Follow us on @GoogleVRP


Thank you to Adam Bacchus, Dirk Göhmann, Sarah Jacobus, Amy Ressler, Martin Straka, Jan Keller, Jon Bottarini


Reducing Security Risks in Open Source Software at Scale: Scorecards Launches V4

Posted by Laurent Simon and Azeem Shaikh, Google Open Source Security Team (GOSST) 


Since our July announcement of Scorecards V2, the Scorecards project—an automated security tool to flag risky supply chain practices in open source projects—has grown steadily to over 40 unique contributors and 18 implemented security checks. Today we are proud to announce the V4 release of Scorecards, with larger scaling, a new security check, and a new Scorecards GitHub Action for easier security automation.

The Scorecards Action is released in partnership with GitHub and is available from GitHub's Marketplace. The Action makes using Scorecards easier than ever: it runs automatically on repository changes to alert developers about risky supply-chain practices. Maintainers can view the alerts on GitHub's code scanning dashboard, which is available for free to public repositories on GitHub.com and via GitHub Advanced Security for private repositories.

Additionally, we have scaled our weekly Scorecards scans to over one million GitHub repositories, and have partnered with the Open Source Insights website for easy user access to the data.

For more details about the release, including the new Dangerous-Workflow security check, visit the OpenSSF's official blog post here.

Apache Log4j Vulnerability

Like many other companies, we’re closely following the multiple CVEs regarding Apache Log4j 2. Our security teams are investigating any potential impact on Google products and services and are focused on protecting our users and customers.

We encourage anyone who manages environments containing Log4j 2 to update to the latest version.

Based on findings in our ongoing investigations, here is our list of product and service updates as of December 17th (CVE-2021-44228 & CVE-2021-45046):

Android is not aware of any impact to the Android Platform or Enterprise. At this time, no update is required for this specific vulnerability, but we encourage our customers to ensure that the latest security updates are applied to their devices.

Chrome OS  releases and infrastructure are not using versions of Log4j affected by the vulnerability.

Chrome Browser releases, infrastructure and admin console are not using versions of Log4j affected by the vulnerability.

Google Cloud has a specific advisory dedicated to updating customers on the status of GCP and Workspace products and services.

Google Marketing Platform, including Google Ads is not using versions of Log4j affected by the vulnerability. This includes Display & Video 360, Search Ads 360, Google Ads, Analytics (360 and free), Optimize 360, Surveys 360 & Tag Manager 360.

YouTube  is not using versions of Log4j affected by the vulnerability.

We will continue to update this advisory with the latest information.

Understanding the Impact of Apache Log4j Vulnerability

Posted by James Wetter and Nicky Ringland, Open Source Insights Team 



More than 35,000 Java packages, amounting to over 8% of the Maven Central repository (the most significant Java package repository), have been impacted by the recently disclosed log4j vulnerabilities (1, 2), with widespread fallout across the software industry. The vulnerabilities allow an attacker to perform remote code execution by exploiting the insecure JNDI lookups feature exposed by the logging library log4j. This exploitable feature was enabled by default in many versions of the library.


This vulnerability has captivated the information security ecosystem since its disclosure on December 9th because of both its severity and widespread impact. As a popular logging tool, log4j is used by tens of thousands of software packages (known as artifacts in the Java ecosystem) and projects across the software industry. User’s lack of visibility into their dependencies and transitive dependencies has made patching difficult; it has also made it difficult to determine the full blast radius of this vulnerability. Using Open Source Insights, a project to help understand open source dependencies, we surveyed all versions of all artifacts in the Maven Central Repository to determine the scope of the issue in the open source ecosystem of JVM based languages, and to track the ongoing efforts to mitigate the affected packages.



How widespread is the log4j vulnerability?

As of December 16, 2021, we found that 35,863 of the available Java artifacts from Maven Central depend on the affected log4j code. This means that more than 8% of all packages on Maven Central have at least one version that is impacted by this vulnerability. (These numbers do not encompass all Java packages, such as directly distributed binaries, but Maven Central is a strong proxy for the state of the ecosystem.)

As far as ecosystem impact goes, 8% is enormous. The average ecosystem impact of advisories affecting Maven Central is 2%, with the median less than 0.1%.





Direct dependencies account for around 7,000 of the affected artifacts, meaning that any of its versions depend upon an affected version of log4j-core or log4j-api, as described in the CVEs. The majority of affected artifacts come from indirect dependencies (that is, the dependencies of one’s own dependencies), meaning log4j is not explicitly defined as a dependency of the artifact, but gets pulled in as a transitive dependency.






What is the current progress in fixing the open source JVM ecosystem?
We counted an artifact as fixed if the artifact had at least one version affected and has released a greater stable version (according to semantic versioning) that is unaffected. An artifact affected by log4j is considered fixed if it has updated to 2.16.0 or removed its dependency on log4j altogether.

At the time of writing, nearly five thousand of the affected artifacts have been fixed. This represents a rapid response and mammoth effort both by the log4j maintainers and the wider community of open source consumers.

That leaves over 30,000 artifacts affected, many of which are dependent on another artifact to patch (the transitive dependency) and are likely blocked.


Why is fixing the JVM ecosystem hard?
Most artifacts that depend on log4j do so indirectly. The deeper the vulnerability is in a dependency chain, the more steps are required for it to be fixed. The following diagram shows a histogram of how deeply an affected log4j package (core or api) first appears in consumers dependency graphs. For greater than 80% of the packages, the vulnerability is more than one level deep, with a majority affected five levels down (and some as many as nine levels down). These packages will require fixes throughout all parts of the tree, starting from the deepest dependencies first.






Another difficulty is caused by ecosystem-level choices in the dependency resolution algorithm and requirement specification conventions.

In the Java ecosystem, it’s common practice to specify “soft” version requirements — exact versions that are used by the resolution algorithm if no other version of the same package appears earlier in the dependency graph. Propagating a fix often requires explicit action by the maintainers to update the dependency requirements to a patched version.

This practice is in contrast to other ecosystems, such as npm, where it’s common for developers to specify open ranges for dependency requirements. Open ranges allow the resolution algorithm to select the most recently released version that satisfies dependency requirements, thereby pulling in new fixes. Consumers can get a patched version on the next build after the patch is available, which propagates up the dependencies quickly. (This approach is not without its drawbacks; pulling in new fixes can also pull in new problems.)


How long will it take for this vulnerability to be fixed across the entire ecosystem?

It’s hard to say. We looked at all publicly disclosed critical advisories affecting Maven packages to get a sense of how quickly other vulnerabilities have been fully addressed. Less than half (48%) of the artifacts affected by a vulnerability have been fixed, so we might be in for a long wait, likely years.

But things are looking promising on the log4j front. After less than a week, 4,620 affected artifacts (~13%) have been fixed. This, more than any other stat, speaks to the massive effort by open source maintainers, information security teams and consumers across the globe.

Where to focus next?

Thanks and congratulations are due to the open source maintainers and consumers who have already upgraded their versions of log4j. As part of our investigation, we pulled together a list of 500 affected packages with some of the highest transitive usage. If you are a maintainer or user helping with the patching effort, prioritizing these packages could maximize your impact and unblock more of the community.

We encourage the open source community to continue to strengthen security in these packages by enabling automated dependency updates and adding security mitigations. Improvements such as these could qualify for financial rewards from the Secure Open Source Rewards program.

You can explore your package dependencies and their vulnerabilities by using Open Source Insights.

Improving OSS-Fuzz and Jazzer to catch Log4Shell

Posted by Jonathan Metzman, Google Open Source Security Team

The discovery of the Log4Shell vulnerability has set the internet on fire. Similar to shellshock and heartbleed, Log4Shell is just the latest catastrophic vulnerability in software that runs the internet. Our mission as the Google Open Source Security Team is to secure the open source libraries the world depends on, such as Log4j. One of our capabilities in this space is OSS-Fuzz, a free fuzzing service that is used by over 500 critical open source projects and has found more than 7,000 vulnerabilities in its lifetime.

We want to empower open source developers to secure their code on their own. Over the next year we will work on better automated detection of non-memory corruption vulnerabilities such as Log4Shell. We have started this work by partnering with the security company Code Intelligence to provide continuous fuzzing for Log4j, as part of OSS-Fuzz. Also as part of this partnership, Code-Intelligence improved their Jazzer fuzzing engine to make it capable of detecting remote JNDI lookups. We have awarded Code Intelligence $25,000 for this effort and will continue to work with them on securing the open source ecosystem.

Caption: OSS-Fuzz and Jazzer finding the Log4Shell Vulnerability



Vulnerabilities like Log4Shell are an eye-opener for the industry in terms of new attack vectors. With OSS-Fuzz and Jazzer, we can now detect this class of vulnerability so that they can be fixed before they become a problem in production code.

Over the past year we have made a number of investments to strengthen the security of critical open source projects, and recently announced our $10 billion commitment to cybersecurity defense including $100 million to support third-party foundations that manage open source security priorities and help fix vulnerabilities.

We appreciate the maintainers, security engineers and incident responders that are working to mitigate Log4j and make our internet ecosystem safer.

Check out our documentation to get started using OSS-Fuzz.

Exploring Container Security: A Storage Vulnerability Deep Dive

Posted by Fabricio Voznika and Mauricio Poppe, Google Cloud 

Kubernetes Security is constantly evolving - keeping pace with enhanced functionality, usability and flexibility while also balancing the security needs of a wide and diverse set of use-cases.

Recently, the GKE Security team discovered a high severity vulnerability that allowed workloads to have access to parts of the host filesystem outside the mounted volumes boundaries. Although the vulnerability was patched back in September we thought it would be beneficial to write up a more in-depth analysis of the issue to share with the community.

We assessed the impact of the vulnerability as described in vulnerability management in open-source Kubernetes and worked closely with the GKE Storage team and the Kubernetes Security Response Committee to find a fix. In this post we’ll give some background on how the subpath storage system works, an overview of the vulnerability, the steps to find the root cause and the fix, and finally some recommendations for GKE and Anthos users.


Kubernetes Filesystems: Intro to Volume Subpath
The vulnerability, CVE-2021-25741, was caused by a race condition during the creation of a subpath bind mount inside a container, and allowed an attacker to gain unauthorized access to the underlying node filesystem and its sensitive files. We’ll describe how that system is supposed to work, and then talk about the vulnerability.

The volume subpath feature in Kubernetes enables sharing a volume in multiple containers inside a pod. For example, we could create a Pod with an InitContainer that creates directories with pre-populated data in a mounted filesystem volume. These directories can then be used by containers in the same Pod by mounting the same volume and optionally specifying a subpath field to limit what's visible inside the container.

While there are some great use cases for this feature, it’s an area that has had vulnerabilities discovered in the past. The kubelet must be extra cautious when handling user-owned subpaths because it operates with privileges in the host. One vulnerability that has been previously discovered involved the creation of a malicious workload where an InitContainer would create a symlink pointing to any location in the host. For example, the InitContainer could mount a volume in /mnt and create a symlink /mnt/attack inside the container pointing to /etc. Later in the Pod lifecycle, another container would attempt to mount the same volume with subpath attack. While preparing the volumes for the container, the kubelet would end up following the symlink to the host’s /etc instead of the container’s /etc, unknowingly exposing the host filesystem to the container. A previous fix made sure that the subpath mount location is resolved and validated to point to a location inside the base volume and that it's not changeable by the user in between the time the path was validated and when the container runtime bind mounts it. This race condition is known as time of check to time of use (TOCTOU) where the subject being validated changes after it has been validated.

These validations and others are summarized in the following container lifecycle sequence diagram.




Volume subpath validations before the container startup

A New TOCTOU Vulnerability: CVE-2021-25741
The latest vulnerability was discovered by performing a symlink attack similar to the one explained above, with the difference being that it constantly swapped the symlink with a directory in a tight loop, using the RENAME_EXCHANGE option with renameat(2). If the timing is just right, the kubelet will see the path as a directory and pass the validation check. Then the mount utility may find that the path is a symlink pointing to the host and follow it, exposing the host filesystem to the container. This is visualized in the following diagram:


The expectation and the attack outcome

The GKE Security and Storage teams worked closely to revise the fix done previously to find a solution. The previous fix takes several steps to ensure that the directory being mounted is safely opened and validated. After the file is opened and validated, the kubelet uses the magic-link path under /proc/[pid]/fd directory for all subsequent operations to ensure the file remains unchanged. However, we found out that all of the efforts were undone by the mount(8) linux utility which was dereferencing the procfs magic-link by default. Once the problem was understood, the fix involved making sure that the mount utility doesn't dereference the magic-links by using the --no-canonicalize flag in the mount command.

The fix is in

Once the problem was well understood, we fixed it inside Kubernetes and quickly released the fix to GKE and Anthos. If GKE auto-upgrade is enabled in your clusters there's no action on your part for this vulnerability, your nodes have already been patched. We strongly recommend that customers utilize auto-upgrades. Auto-upgrade gives peace of mind that your clusters are running with the latest patches.

GKE released a Google Kubernetes Engine security bulletin on this vulnerability, which detailed what customers can do to immediately remediate this issue across GKE and Anthos. We also provided guidance to customers who manually manage their node versions, ensuring that fixed releases were available in every region for our Static and Release Channels.

Moving forward
Google continues to invest heavily in the security of GKE and Kubernetes. We encourage users interested in finding vulnerabilities to participate in the Kubernetes bug bounty program and in the Google Vulnerability Rewards Program (VRP) which was recently expanded to cover GKE vulnerabilities. For the latest guidance on security issues, please follow our GKE Security Bulletins.

ClusterFuzzLite: Continuous fuzzing for all

Posted by Jonathan Metzman, Google Open Source Security Team


In recent years, continuous fuzzing has become an essential part of the software development lifecycle. By feeding unexpected or random data into a program, fuzzing catches bugs that would otherwise slip through the most thorough manual checks and provides coverage that would take staggering human effort to replicate. NIST’s guidelines for software verification, recently released in response to the White House Executive Order on Improving the Nation’s Cybersecurity, specify fuzzing among the minimum standard requirements for code verification.

Today, we are excited to announce ClusterFuzzLite, a continuous fuzzing solution that runs as part of CI/CD workflows to find vulnerabilities faster than ever before. With just a few lines of code, GitHub users can integrate ClusterFuzzLite into their workflow and fuzz pull requests to catch bugs before they are committed, enhancing the overall security of the software supply chain.

Since its release in 2016, over 500 critical open source projects have integrated into Google’s OSS-Fuzz program, resulting in over 6,500 vulnerabilities and 21,000 functional bugs being fixed. ClusterFuzzLite goes hand-in-hand with OSS-Fuzz, by catching regression bugs much earlier in the development process.

Large projects including systemd and curl are already using ClusterFuzzLite during code review, with positive results. According to Daniel Stenberg, author of curl, “When the human reviewers nod and have approved the code and your static code analyzers and linters can't detect any more issues, fuzzing is what takes you to the next level of code maturity and robustness. OSS-Fuzz and ClusterFuzzLite help us maintain curl as a quality project, around the clock, every day and every commit.”

With the release of ClusterFuzzLite, any project can integrate this essential testing standard and benefit from fuzzing. ClusterFuzzLite offers many of the same features as ClusterFuzz, such as continuous fuzzing, sanitizer support, corpus management, and coverage report generation. Most importantly, it’s easy to set up and works with closed source projects, making ClusterFuzzLite a convenient option for any developer who wants to fuzz their software.



 


With ClusterFuzzLite, fuzzing is no longer just an idealized "bonus" round of testing for those who have access to it, but a critical must-have step that everyone can use continuously on every software project. By finding and preventing bugs before they enter the codebase we can build a more secure software ecosystem.

To learn more, check out the ClusterFuzzLite documentation. ClusterFuzzLite currently supports GitHub Actions, Google Cloud Build and Prow. We built this with CI system extensibility in mind, and adding support for other CI systems is straightforward. Please contact us if you’re interested in contributing support, or have any questions, feedback or feature requests.