Author Archives: Kimberly Samra

Google Public DNS’s approach to fight against cache poisoning attacks



The Domain Name System (DNS) is a fundamental protocol used on the Internet to translate human-readable domain names (e.g., www.example.com) into numeric IP addresses (e.g., 192.0.2.1) so that devices and servers can find and communicate with each other. When a user enters a domain name in their browser, the DNS resolver (e.g. Google Public DNS) locates the authoritative DNS nameservers for the requested name, and queries one or more of them to obtain the IP address(es) to return to the browser.

When DNS was launched in the early 1980s as a trusted, content-neutral infrastructure, security was not yet a pressing concern, however, as the Internet grew DNS became vulnerable to various attacks. In this post, we will look at DNS cache poisoning attacks and how Google Public DNS addresses the risks associated with them.

DNS Cache Poisoning Attacks

DNS lookups in most applications are forwarded to a caching resolver (which could be local or an open resolver like. Google Public DNS). The path from a client to the resolver is usually on a local network or can be protected using encrypted transports like DoH, DoT. The resolver queries authoritative DNS servers to obtain answers for user queries. This communication primarily occurs over UDP, an insecure connectionless protocol, in which messages can be easily spoofed including the source IP address. The content of DNS queries may be sufficiently predictable that even an off-path attacker can, with enough effort, forge responses that appear to be from the queried authoritative server. This response will be cached if it matches the necessary fields and arrives before the authentic response. This type of attack is called a cache poisoning attack, which can cause great harm once successful. According to RFC 5452, the probability of success is very high without protection. Forged DNS responses can lead to denial of service, or may even compromise application security. For an excellent introduction to cache poisoning attacks, please see “An Illustrated Guide to the Kaminsky DNS Vulnerability”.

Cache poisoning mitigations in Google Public DNS

Improving DNS security has been a goal of Google Public DNS since our launch in 2009. We take a multi-pronged approach to protect users against DNS cache-poisoning attacks. There is no silver bullet or countermeasure that entirely solves the problem, but in combination they make successful attacks substantially more difficult.


RFC 5452 And DNS Cookies

We have implemented the basic countermeasures outlined in RFC 5452 namely randomizing query source ports and query IDs. But these measures alone are not sufficient (see page 8 of our OARC 38 presentation).

We have therefore also implemented support for RFC 7873 (DNS Cookies) which can make spoofing impractical if it’s supported by the authoritative server. Measurements indicate that the DNS Cookies do not provide sufficient coverage, even though around 40% of nameservers by IP support DNS Cookies, these account for less than 10% of overall query volume. In addition, many non-compliant nameservers return incorrect or ambiguous responses for queries with DNS Cookies, which creates further deployment obstacles. For now, we’ve enabled DNS Cookies through manual configuration, primarily for selected TLD zones.

Case Randomization (0x20)

The query name case randomization mechanism, originally proposed in a March 2008 draft “Use of Bit 0x20 in DNS Labels to Improve Transaction Identity”, however, is highly effective, because all but a small minority of nameservers are compatible with query name case randomization. We have been performing case randomization of query names since 2009 to a small set of chosen nameservers that handle only a minority of our query volume. 

In 2022 we started work on enabling case randomization by default, which when used, the query name in the question section is randomized and the DNS server’s response is expected to match the case-randomized query name exactly in the request. For example, if “ExaMplE.CoM” is the name sent in the request, the name in the question section of the response must also be “ExaMplE.CoM” rather than, e.g., “example.com.” Responses that fail to preserve the case of the query name may be dropped as potential cache poisoning attacks (and retried over TCP).

We are happy to announce that we’ve already enabled and deployed this feature globally by default. It covers over 90% of our UDP traffic to nameservers, significantly reducing the risk of cache poisoning attacks.

Meanwhile, we maintain an exception list and implement fallback mechanisms to prevent potential issues with non-conformant nameservers. However we strongly recommend that nameserver implementations preserve the query case in the response.

DNS-over-TLS

In addition to case randomization, we’ve deployed DNS-over-TLS to authoritative nameservers (ADoT), following procedures described in RFC 9539 (Unilateral Opportunistic Deployment of Encrypted Recursive-to-Authoritative DNS). Real world measurements show that ADoT has a higher success rate and comparable latency to UDP. And ADoT is in use for around 6% of egress traffic. At the cost of some CPU and memory, we get both security and privacy for nameserver queries without DNS compliance issues.

Summary

Google Public DNS takes security of our users seriously. Through multiple countermeasures to cache poisoning attacks, we aim to provide a more secure and reliable DNS resolution service, enhancing the overall Internet experience for users worldwide. With the measures described above we are able to provide protection against passive attacks for over 90% of authoritative queries.


To enhance DNS security, we recommend that DNS server operators support one or more of the  security mechanisms described here. We are also working with the DNS community to improve DNS security. Please see our presentations at DNS-OARC 38 and 40 for more technical details.

Secure by Design: Google’s Perspective on Memory Safety


UN Cybercrime Treaty Could Endanger Web Security



This week, the United Nations convened member states to continue its years-long negotiations on the UN Cybercrime Treaty, titled “Countering the Use of Information and Communications Technologies for Criminal Purposes.” 




As more aspects of our lives intersect with the digital sphere, law enforcement around the world has increasingly turned to electronic evidence to investigate and disrupt criminal activity. Google takes the threat of cybercrime very seriously, and dedicates significant resources to combating it. When governments send Google legal orders to disclose user data in connection with their investigations, we carefully review those orders to make sure they satisfy applicable laws, international norms, and Google’s policies. We also regularly report the number of these orders in our Transparency Report




To ensure that transnational legal demands are issued consistent with rule of law, we have long called for an international framework for digital evidence that includes robust due process protections, respects human rights (including the right to free expression), and aligns with existing international norms. This is particularly important in the case of transnational criminal investigations, where the legal protections  in one jurisdiction may not align with those in others. 




Such safeguards aren’t just important to ensuring free expression and human rights, they are also critical to protecting web security. Too often, as we know well from helping stand up the Security Researcher Legal Defense Fund, individuals working to advance cybersecurity for the public good end up facing criminal charges. The Cybercrime Treaty should not criminalize the work of legitimate cybersecurity researchers and penetration testers, which is designed to protect individual systems and  the web as a whole.  




UN Member States have an opportunity to strengthen global cybersecurity by adopting a treaty that encourages the criminalization of the most egregious and systemic activities — on which all parties can agree — while adopting a framework for sharing digital evidence that is transparent, grounded in the rule of law, based on pre-existing international frameworks like the Universal Declaration on Human Rights, and aligned with principles of necessity and proportionality. At the same time, Member States should avoid attempts to criminalize activities that raise significant freedom of expression issues, or that actually undercut the treaty’s goal of reducing cybercrime. That will require strengthening critical guardrails and protections.  




We urge Member States to heed calls from civil society groups to address critical gaps in the Treaty and revise the text to protect users and security professionals — not endanger the security of the web.  

Scaling security with AI: from detection to solution


The AI world moves fast, so we’ve been hard at work keeping security apace with recent advancements. One of our approaches, in alignment with Google’s Safer AI Framework (SAIF), is using AI itself to automate and streamline routine and manual security tasks, including fixing security bugs. Last year we wrote about our experiences using LLMs to expand vulnerability testing coverage, and we’re excited to share some updates. 




Today, we’re releasing our fuzzing framework as a free, open source resource that researchers and developers can use to improve fuzzing’s bug-finding abilities. We’ll also show you how we’re using AI to speed up the bug patching process. By sharing these experiences, we hope to spark new ideas and drive innovation for a stronger ecosystem security.



Update: AI-powered vulnerability discovery

Last August, we announced our framework to automate manual aspects of fuzz testing (“fuzzing”) that often hindered open source maintainers from fuzzing their projects effectively. We used LLMs to write project-specific code to boost fuzzing coverage and find more vulnerabilities. Our initial results on a subset of projects in our free OSS-Fuzz service were very promising, with code coverage increased by 30% in one example. Since then, we’ve expanded our experiments to more than 300 OSS-Fuzz C/C++ projects, resulting in significant coverage gains across many of the project codebases. We’ve also improved our prompt generation and build pipelines, which has increased code line coverage by up to 29% in 160 projects. 


How does that translate to tangible security improvements? So far, the expanded fuzzing coverage offered by LLM-generated improvements allowed OSS-Fuzz to discover two new vulnerabilities in cJSON and libplist, two widely used projects that had already been fuzzed for years. As always, we reported the vulnerabilities to the project maintainers for patching. Without the completely LLM-generated code, these two vulnerabilities could have remained undiscovered and unfixed indefinitely. 


And more: AI-powered vulnerability fixing

Fuzzing is fantastic for finding bugs, but for security to improve, those bugs also need to be patched. It’s long been an industry-wide struggle to find the engineering hours needed to patch open bugs at the pace that they are uncovered, and triaging and fixing bugs is a significant manual toll on project maintainers. With continued improvements in using LLMs to find more bugs, we need to keep pace in creating similarly automated solutions to help fix those bugs. We recently announced an experiment doing exactly that: building an automated pipeline that intakes vulnerabilities (such as those caught by fuzzing), and prompts LLMs to generate fixes and test them before selecting the best for human review.




This AI-powered patching approach resolved 15% of the targeted bugs, leading to significant time savings for engineers. The potential of this technology should apply to most or all categories throughout the software development process. We’re optimistic that this research marks a promising step towards harnessing AI to help ensure more secure and reliable software.




Try it out

Since we’ve now open sourced our framework to automate manual aspects of fuzzing, any researcher or developer can experiment with their own prompts to test the effectiveness of fuzz targets generated by LLMs (including Google’s VertexAI or their own fine-tuned models) and measure the results against OSS-Fuzz C/C++ projects. We also hope to encourage research collaborations and to continue seeing other work inspired by our approach, such as Rust fuzz target generation


If you’re interested in using LLMs to patch bugs, be sure to read our paper on building an AI-powered patching pipeline. You’ll find a summary of our own experiences, some unexpected data about LLM’s abilities to patch different types of bugs, and guidance for building pipelines in your own organizations. 

Improving Text Classification Resilience and Efficiency with RETVec



Systems such as Gmail, YouTube and Google Play rely on text classification models to identify harmful content including phishing attacks, inappropriate comments, and scams. These types of texts are harder for machine learning models to classify because bad actors rely on adversarial text manipulations to actively attempt to evade the classifiers. For example, they will use homoglyphs, invisible characters, and keyword stuffing to bypass defenses. 




To help make text classifiers more robust and efficient, we’ve developed a novel, multilingual text vectorizer called RETVec (Resilient & Efficient Text Vectorizer) that helps models achieve state-of-the-art classification performance and drastically reduces computational cost. Today, we’re sharing how RETVec has been used to help protect Gmail inboxes.




Strengthening the Gmail Spam Classifier with RETVec


Figure 1. RETVec-based Gmail Spam filter improvements.




Over the past year, we battle-tested RETVec extensively inside Google to evaluate its usefulness and found it to be highly effective for security and anti-abuse applications. In particular, replacing the Gmail spam classifier’s previous text vectorizer with RETVec allowed us to improve the spam detection rate over the baseline by 38% and reduce the false positive rate by 19.4%. Additionally, using RETVec reduced the TPU usage of the model by 83%, making the RETVec deployment one of the largest defense upgrades in recent years. RETVec achieves these improvements by sporting a very lightweight word embedding model (~200k parameters), allowing us to reduce the Transformer model’s size at equal or better performance, and having the ability to split the computation between the host and TPU in a network and memory efficient manner.




RETVec Benefits

RETVec achieves these improvements by combining a novel, highly-compact character encoder, an augmentation-driven training regime, and the use of metric learning. The architecture details and benchmark evaluations are available in our NeurIPS 2023 paper and we open-source RETVec on Github.




Due to its novel architecture, RETVec works out-of-the-box on every language and all UTF-8 characters without the need for text preprocessing, making it the ideal candidate for on-device, web, and large-scale text classification deployments. Models trained with RETVec exhibit faster inference speed due to its compact representation. Having smaller models reduces computational costs and decreases latency, which is critical for large-scale applications and on-device models.




Figure 1. RETVec architecture diagram.





Models trained with RETVec can be seamlessly converted to TFLite for mobile and edge devices, as a result of a native implementation in TensorFlow Text. For web application model deployment, we provide a TensorflowJS layer implementation that is available on Github and you can check out a demo web page running a RETVec-based model.




Figure 2.  Typo resilience of text classification models trained from scratch using different vectorizers.




RETVec is a novel open-source text vectorizer that allows you to build more resilient and efficient server-side and on-device text classifiers. The Gmail spam filter uses it to help protect Gmail inboxes against malicious emails.





If you would like to use RETVec for your own use cases or research, we created a tutorial to help you get started.







This research was conducted by Elie Bursztein, Marina Zhang, Owen Vallis, Xinyu Jia, and Alexey Kurakin. We would like to thank Gengxin Miao, Brunno Attorre, Venkat Sreepati, Lidor Avigad, Dan Givol, Rishabh Seth and Melvin Montenegro and all the Googlers who contributed to the project.


Two years later: a baseline that drives up security for the industry


Nearly half of third-parties fail to meet two or more of the Minimum Viable Secure Product controls. Why is this a problem? Because "98% of organizations have a relationship with at least one third-party that has experienced a breach in the last 2 years."

In this post, we're excited to share the latest improvements to the Minimum Viable Secure Product (MVSP) controls. We'll also shed light on how adoption of MVSP has helped Google improve its security processes, and hope this example will help motivate third-parties to increase their adoption of MVSP controls and thus improve product security across the industry.

About MVSP

In October 2021, Google publicly launched MVSP alongside launch partners. Our original goal remains unchanged: to provide a vendor-neutral application security baseline, designed to eliminate overhead, complexity, and confusion in the end-to-end process of onboarding third-party products and services. It covers themes such as procurement, security assessment, and contract negotiation.




Improvements since launch

As part of MVSP’s annual control review, and our core philosophy of evolution over revolution, the working group sought input from the broader security community to ensure MVSP maintains a balance between security and achievability.

As a result of these discussions, we launched updated controls. Key changes include: expanded guidance around external vulnerability reporting to protect bug hunters, and discouraging additional costs for access to basic security features – inline with CISA’s "Secure-by-Design" principles.

In 2022, we developed guidance on build process security based on SLSA, to reflect the importance of supply chain security and integrity.

From an organizational perspective, in the two years since launching, we've seen the community around MVSP continue to expand. The working group has grown to over 20 global members, helping to diversify voices and broaden expertise. We've also had the opportunity to present and discuss the program with a number of key groups, including an invitation to present at the United Nations International Computing Centre – Common Secure Conference.

Google at the UNICC conference in Valencia, Spain

How Google uses MVSP

Since its inception, Google has looked to integrate improvements to our own processes using MVSP as a template. Two years later, we can clearly see the impact through faster procurement processes, streamlined contract negotiations, and improved data-driven decision making.

Highlights

  • After implementing MVSP into key areas of Google's third-party life-cycle, we've observed a 68% reduction in the time required for third-parties to complete assessment process.

  • By embedding MVSP into select procurement processes, Google has increased data-driven decision making in earlier phases of the cycle.

  • Aligning our Information Protection Addendum’s safeguards with MVSP has significantly improved our third-party privacy and security risk management processes.

You use MVSP to enhance your software or procurement processes by reviewing some common use-cases and adopting them into your third-party risk management and/or contracting workflows .

What's next?

We're invested in helping the industry manage risk posture through continuous improvement, while increasing the minimum bar for product security across the industry.

By making MVSP available to the wider industry, we are helping to create a solid foundation for growing the maturity level of products and services. Google has benefited from driving security and safety improvements through the use of leveled sets of requirements. We expect the same to be true across the wider industry.


We've seen success, but there is still work to be done. Based on initial observations, as mentioned above, 48% of third-parties fail to meet two or more of the Minimum Viable Secure Product controls.


As an industry, we can't stand still when it comes to product security. Help us raise the minimum bar for application security by adopting MVSP and ensuring we as an industry don’t accept anything less than a strong security baseline that works for the wider industry.

Acknowledgements

Google and the MVSP working group would like to thank those who have supported and contributed since its inception. If you'd like to get involved or provide feedback, please reach out.



Thank you to Chris John Riley, Gabor Acs-Kurucz, Michele Chubirka, Anna Hupa, Dirk Göhmann and Kaan Kivilcim from the Google MVSP Group for their contributions to this post.


Increasing transparency in AI security


New AI innovations and applications are reaching consumers and businesses on an almost-daily basis. Building AI securely is a paramount concern, and we believe that Google’s Secure AI Framework (SAIF) can help chart a path for creating AI applications that users can trust. Today, we’re highlighting two new ways to make information about AI supply chain security universally discoverable and verifiable, so that AI can be created and used responsibly. 




The first principle of SAIF is to ensure that the AI ecosystem has strong security foundations. In particular, the software supply chains for components specific to AI development, such as machine learning models, need to be secured against threats including model tampering, data poisoning, and the production of harmful content




Even as machine learning and artificial intelligence continue to evolve rapidly, some solutions are now within reach of ML creators. We’re building on our prior work with the Open Source Security Foundation to show how ML model creators can and should protect against ML supply chain attacks by using SLSA and Sigstore.



Supply chain security for ML


For supply chain security of conventional software (software that does not use ML), we usually consider questions like:


  • Who published the software? Are they trustworthy? Did they use safe practices?
  • For open source software, what was the source code?
  • What dependencies went into building that software?
  • Could the software have been replaced by a tampered version following publication? Could this have occurred during build time?




All of these questions also apply to the hundreds of free ML models that are available for use on the internet. Using an ML model means trusting every part of it, just as you would any other piece of software. This includes concerns such as:



  • Who published the model? Are they trustworthy? Did they use safe practices?
  • For open source models, what was the training code?
  • What datasets went into training that model?
  • Could the model have been replaced by a tampered version following publication? Could this have occurred during training time?




We should treat tampering of ML models with the same severity as we treat injection of malware into conventional software. In fact, since models are programs, many allow the same types of arbitrary code execution exploits that are leveraged for attacks on conventional software. Furthermore, a tampered model could leak or steal data, cause harm from biases, or spread dangerous misinformation. 




Inspection of an ML model is insufficient to determine whether bad behaviors were injected. This is similar to trying to reverse engineer an executable to identify malware. To protect supply chains at scale, we need to know how the model or software was created to answer the questions above.



Solutions for ML supply chain security


In recent years, we’ve seen how providing public and verifiable information about what happens during different stages of software development is an effective method of protecting conventional software against supply chain attacks. This supply chain transparency offers protection and insights with:



  • Digital signatures, such as those from Sigstore, which allow users to verify that the software wasn’t tampered with or replaced
  • Metadata such as SLSA provenance that tell us what’s in software and how it was built, allowing consumers to ensure license compatibility, identify known vulnerabilities, and detect more advanced threats





Together, these solutions help combat the enormous uptick in supply chain attacks that have turned every step in the software development lifecycle into a potential target for malicious activity.




We believe transparency throughout the development lifecycle will also help secure ML models, since ML model development follows a similar lifecycle as for regular software artifacts:




Similarities between software development and ML model development





An ML training process can be thought of as a “build:” it transforms some input data to some output data. Similarly, training data can be thought of as a “dependency:” it is data that is used during the build process. Because of the similarity in the development lifecycles, the same software supply chain attack vectors that threaten software development also apply to model development: 





Attack vectors on ML through the lens of the ML supply chain




Based on the similarities in development lifecycle and threat vectors, we propose applying the same supply chain solutions from SLSA and Sigstore to ML models to similarly protect them against supply chain attacks.



Sigstore for ML models



Code signing is a critical step in supply chain security. It identifies the producer of a piece of software and prevents tampering after publication. But normally code signing is difficult to set up—producers need to manage and rotate keys, set up infrastructure for verification, and instruct consumers on how to verify. Often times secrets are also leaked since security is hard to get right during the process.




We suggest bypassing these challenges by using Sigstore, a collection of tools and services that make code signing secure and easy. Sigstore allows any software producer to sign their software by simply using an OpenID Connect token bound to either a workload or developer identity—all without the need to manage or rotate long-lived secrets.




So how would signing ML models benefit users? By signing models after training, we can assure users that they have the exact model that the builder (aka “trainer”) uploaded. Signing models discourages model hub owners from swapping models, addresses the issue of a model hub compromise, and can help prevent users from being tricked into using a bad model. 




Model signatures make attacks similar to PoisonGPT detectable. The tampered models will either fail signature verification or can be directly traced back to the malicious actor. Our current work to encourage this industry standard includes:




  • Having ML frameworks integrate signing and verification in the model save/load APIs
  • Having ML model hubs add a badge to all signed models, thus guiding users towards signed models and incentivizing signatures from model developers
  • Scaling model signing for LLMs 




SLSA for ML Supply Chain Integrity



Signing with Sigstore provides users with confidence in the models that they are using, but it cannot answer every question they have about the model. SLSA goes a step further to provide more meaning behind those signatures. 




SLSA (Supply-chain Levels for Software Artifacts) is a specification for describing how a software artifact was built. SLSA-enabled build platforms implement controls to prevent tampering and output signed provenance describing how the software artifact was produced, including all build inputs. This way, SLSA provides trustworthy metadata about what went into a software artifact.




Applying SLSA to ML could provide similar information about an ML model’s supply chain and address attack vectors not covered by model signing, such as compromised source control, compromised training process, and vulnerability injection. Our vision is to include specific ML information in a SLSA provenance file, which would help users spot an undertrained model or one trained on bad data. Upon detecting a vulnerability in an ML framework, users can quickly identify which models need to be retrained, thus reducing costs.




We don’t need special ML extensions for SLSA. Since an ML training process is a build (shown in the earlier diagram), we can apply the existing SLSA guidelines to ML training. The ML training process should be hardened against tampering and output provenance just like a conventional build process. More work on SLSA is needed to make it fully useful and applicable to ML, particularly around describing dependencies such as datasets and pretrained models.  Most of these efforts will also benefit conventional software.




For models training on pipelines that do not require GPUs/TPUs, using an existing, SLSA-enabled build platform is a simple solution. For example, Google Cloud Build, GitHub Actions, or GitLab CI are all generally available SLSA-enabled build platforms. It is possible to run an ML training step on one of these platforms to make all of the built-in supply chain security features available to conventional software.



How to do model signing and SLSA for ML today



By incorporating supply chain security into the ML development lifecycle now, while the problem space is still unfolding, we can jumpstart work with the open source community to establish industry standards to solve pressing problems. This effort is already underway and available for testing.  




Our repository of tooling for model signing and experimental SLSA provenance support for smaller ML models is available now. Our future ML framework and model hub integrations will be released in this repository as well. 




We welcome collaboration with the ML community and are looking forward to reaching consensus on how to best integrate supply chain protection standards into existing tooling (such as Model Cards). If you have feedback or ideas, please feel free to open an issue and let us know. 

Google’s reward criteria for reporting bugs in AI products


In September, we shared how we are implementing the voluntary AI commitments that we and others in industry made at the White House in July. One of the most important developments involves expanding our existing Bug Hunter Program to foster third-party discovery and reporting of issues and vulnerabilities specific to our AI systems. Today, we’re publishing more details on these new reward program elements for the first time. Last year we issued over $12 million in rewards to security researchers who tested our products for vulnerabilities, and we expect today’s announcement to fuel even greater collaboration for years to come. 




What’s in scope for rewards 

In our recent AI Red Team report, we identified common tactics, techniques, and procedures (TTPs) that we consider most relevant and realistic for real-world adversaries to use against AI systems. The following table incorporates shared learnings from Google’s AI Red Team exercises to help the research community better understand what’s in scope for our reward program. We're detailing our criteria for AI bug reports to assist our bug hunting community in effectively testing the safety and security of AI products. Our scope aims to facilitate testing for traditional security vulnerabilities as well as risks specific to AI systems. It is important to note that reward amounts are dependent on severity of the attack scenario and the type of target affected (go here for more information on our reward table). 





Category

Attack Scenario

Guidance

Prompt Attacks: Crafting adversarial prompts that allow an adversary to influence the behavior of the model, and hence the output in ways that were not intended by the application.

Prompt injections that are invisible to victims and change the state of the victim's account or or any of their assets.

In Scope

Prompt injections into any tools in which the response is used to make decisions that directly affect victim users.

In Scope

Prompt or preamble extraction in which a user is able to extract the initial prompt used to prime the model only when sensitive information is present in the extracted preamble.

In Scope

Using a product to generate violative, misleading, or factually incorrect content in your own session: e.g. 'jailbreaks'. This includes 'hallucinations' and factually inaccurate responses. Google's generative AI products already have a dedicated reporting channel for these types of content issues.

Out of Scope

Training Data Extraction: Attacks that are able to successfully reconstruct verbatim training examples that contain sensitive information. Also called membership inference.


Training data extraction that reconstructs items used in the training data set that leak sensitive, non-public information.

In Scope

Extraction that reconstructs nonsensitive/public information.

Out of Scope

Manipulating Models: An attacker able to covertly change the behavior of a model such that they can trigger pre-defined adversarial behaviors.


Adversarial output or behavior that an attacker can reliably trigger via specific input in a model owned and operated by Google ("backdoors"). Only in-scope when a model's output is used to change the state of a victim's account or data. 

In Scope

Attacks in which an attacker manipulates the training data of the model to influence the model’s output in a victim's session according to the attacker’s preference. Only in-scope when a model's output is used to change the state of a victim's account or data. 

In Scope

Adversarial Perturbation: Inputs that are provided to a model that results in a deterministic, but highly unexpected output from the model.

Contexts in which an adversary can reliably trigger a misclassification in a security control that can be abused for malicious use or adversarial gain. 

In Scope

Contexts in which a model's incorrect output or classification does not pose a compelling attack scenario or feasible path to Google or user harm.

Out of Scope

Model Theft / Exfiltration: AI models often include sensitive intellectual property, so we place a high priority on protecting these assets. Exfiltration attacks allow attackers to steal details about a model such as its architecture or weights.

Attacks in which the exact architecture or weights of a confidential/proprietary model are extracted.

In Scope

Attacks in which the architecture and weights are not extracted precisely, or when they're extracted from a non-confidential model.

Out of Scope

If you find a flaw in an AI-powered tool other than what is listed above, you can still submit, provided that it meets the qualifications listed on our program page.


A bug or behavior that clearly meets our qualifications for a valid security or abuse issue.


In Scope

Using an AI product to do something potentially harmful that is already possible with other tools. For example, finding a vulnerability in open source software (already possible using publicly-available static analysis tools) and producing the answer to a harmful question when the answer is already available online.

Out of Scope

As consistent with our program, issues that we already know about are not eligible for reward.

Out of Scope

Potential copyright issues: findings in which products return content appearing to be copyright-protected. Google's generative AI products already have a dedicated reporting channel for these types of content issues.

Out of Scope





Conclusion 

We look forward to continuing our work with the research community to discover and fix security and abuse issues in our AI-powered features. If you find a qualifying issue, please go to our Bug Hunter website to send us your bug report and–if the issue is found to be valid–be rewarded for helping us keep our users safe.

Joint Industry statement of support for Consumer IoT Security Principles



Last week at Singapore International Cyber Week and the ETSI Security Conferences, the international community gathered together to discuss cybersecurity hot topics of the day. Amidst a number of important cybersecurity discussions, we want to highlight progress on connected device security demonstrated by  joint industry principles for IoT security transparency. The future of connected devices offers tremendous potential for innovation and quality of life improvements. Putting a spotlight on consumer IoT security is a key aspect of achieving these benefits. Marketplace competition can be an important driver of security improvements, with consumers empowered and motivated to make informed purchasing decisions based on device security. 

As with other IoT security transparency initiatives globally, it’s great to see this topic being covered at both conferences this week. The below IoT security labeling principles are aimed at helping to improve consumer awareness and to foster marketplace competition based on security.

To help consumers make an informed purchase decision they should receive clear, consistent, and actionable information about the security of the device (e.g. security support period, authentication support, cryptographic assurance) before purchase - a communication and transparency mechanism commonly referred to as “a label” or “labeling,” although the communication is not merely a printed sticker on physical product packaging. While an IoT label will not solve the problem of IoT security on its own, transparency can both help educate consumers and also facilitate the coordination of security responsibilities between all of the components in a connected device ecosystem.

Our goal is to strengthen the security of IoT devices and ecosystems to protect individuals and organizations, and to unleash the full future benefit of IoT. Security labeling programs can support consumer purchase decisions that drive security improvements, but only if the label is credible, actionable, and easily understood. We are hopeful that the public sector and industry can work together to drive harmonized policies that achieve this goal. 

Signed,

Google

ARM

HackerOne

Keysight

NXP

OpenPolicy

Rapid7

Schlage

Silicon Labs

Assa Abloy

Joint Industry statement of support for Consumer IoT Security Principles



Last week at Singapore International Cyber Week and the ETSI Security Conferences, the international community gathered together to discuss cybersecurity hot topics of the day. Amidst a number of important cybersecurity discussions, we want to highlight progress on connected device security demonstrated by  joint industry principles for IoT security transparency. The future of connected devices offers tremendous potential for innovation and quality of life improvements. Putting a spotlight on consumer IoT security is a key aspect of achieving these benefits. Marketplace competition can be an important driver of security improvements, with consumers empowered and motivated to make informed purchasing decisions based on device security. 

As with other IoT security transparency initiatives globally, it’s great to see this topic being covered at both conferences this week. The below IoT security labeling principles are aimed at helping to improve consumer awareness and to foster marketplace competition based on security.

To help consumers make an informed purchase decision they should receive clear, consistent, and actionable information about the security of the device (e.g. security support period, authentication support, cryptographic assurance) before purchase - a communication and transparency mechanism commonly referred to as “a label” or “labeling,” although the communication is not merely a printed sticker on physical product packaging. While an IoT label will not solve the problem of IoT security on its own, transparency can both help educate consumers and also facilitate the coordination of security responsibilities between all of the components in a connected device ecosystem.

Our goal is to strengthen the security of IoT devices and ecosystems to protect individuals and organizations, and to unleash the full future benefit of IoT. Security labeling programs can support consumer purchase decisions that drive security improvements, but only if the label is credible, actionable, and easily understood. We are hopeful that the public sector and industry can work together to drive harmonized policies that achieve this goal. 

Signed,

Google

ARM

HackerOne

Keysight

NXP

OpenPolicy

Rapid7

Schlage

Silicon Labs

Assa Abloy