Tag Archives: Google Cloud Platform

Monitor your GCP environment with Cloud Security Command Center

By Andy Chang, Senior Product Manager

Last week, we announced the release of Cloud Security Command CenterAlpha (Cloud SCC), a new security data analysis and monitoring platform for Google Cloud Platform (GCP). Cloud SCC, now available in alpha, helps enterprises gather security information, identify threats and take action on them.

As the use of cloud services continues to grow, clear visibility into the security status of an organization’s cloud services and resources is more important than ever. Businesses need the right data and actionable insights to stop threats before security incidents do any damage. Cloud SCC takes inventory of your cloud assets, flags unwanted changes to those assets and uses a number of unique detectors to identify risky areas in your environment. Its findings are populated into a single, centralized dashboard and data platform so that you can quickly get a read on the security health of your cloud applications and data.
Cloud SCC aggregates security information in a single, centralized dashboard
In this blog post, we’ll take a deeper look into the capabilities and features of Cloud Security Command Center.

Gain visibility into your cloud services and resources


Cloud SCC gives enterprises consolidated visibility into their cloud assets across App Engine, Compute Engine, Cloud Storage, and Datastore. Using asset inventory, you can view resources for the entire GCP organization or just for particular projects. Cloud SCC performs ongoing discovery scans which allows you to see asset history to understand exactly what changed in your environment and act on unauthorized modifications.
Cloud SCC gives you broad visibility cloud assets at the org and project level
Cloud SCC also features security “marks” that let you personalize how your security information is displayed, organized and managed in order to meet the unique requirements of your organization. With security marks, you can annotate your assets and then search, select, or filter using the mark—for example, you can filter out projects that you group together using the same mark.

Leverage powerful security insights from Google and leading security partners


Cloud SCC generates curated insights that provide you with a unique view of threats to your cloud assets. For example, security teams can answer questions like “Which cloud storage buckets contain PII?”, “Do I have any buckets that are open to the Internet?” and “Which cloud applications are vulnerable to XSS vulnerabilities?” With increasingly frequent reports of sensitive data being inadvertently exposed, gaining visibility into these key risk areas is especially important for enterprises. Cloud SCC integrates with Google Cloud security tools and leading security partners to give you these valuable security insights.

Detection from Google

Cloud SCC integrates with a number of Google Cloud security tools. With information from the DLP API, you can find out which storage buckets contain sensitive and regulated data, help prevent unintended exposure, and ensure access is based on need-to-know. You can also pull in information from Cloud Security Scanner which uncovers common vulnerabilities such as cross-site-scripting (XSS) and Flash injection that put your Google App Engine applications at risk. Using Forseti, an open source security toolkit for GCP, you can identify misconfigured access control policies and respond right away.
These Cloud SCC views show permission changes detected by Forseti, an open source GCP security toolkit
Administrators can also identify threats like botnets, cryptocurrency mining, and suspicious network traffic in your projects and virtual machine (VM) instances with built-in anomaly detection developed by the Google security team.
Cloud SCC features built-in anomaly detection from Google to identify threats to your cloud environment
This Cloud SCC "card" shows sensitive data uncovered by the DLP API
Detection from security partners

Using Cloud SCC, you can leverage intelligence from your existing security tools such as Cloudflare, CrowdStrike, Dome9, Palo Alto Networks, Qualys, and RedLock into Cloud Security Command Center to help detect DDoS attacks, compromised endpoints, compliance policy violations, network attacks, and instance vulnerabilities and threats. Our partner solutions cover a broad set of enterprise security needs and we’ll continue to add new partnerships to our network in the future.

Take advantage of an open and flexible platform


Cloud Security Command Center features a REST API which gives you the flexibility to work with your existing security systems and workflows. Using the API, enterprises can easily integrate the full range of their own threat detection capabilities—once the data sources are forwarded to Cloud Security Command Center, they can be viewed just like the Google-provided Command Center detectors. In addition, you can take advantage of the Pub/Sub notification integration to receive Cloud SCC alerts via Gmail, SMS, and Jira.

Try Cloud Security Command Center today


We’re excited to bring the Cloud SCC security monitoring platform to the suite of GCP security services. To learn more, check out the product documentation, or get started today by signing up for the Cloud SCC alpha program.

Kubernetes 1.10: an insider take on what’s new

By Craig Box, Cloud Native Advocacy Lead, Google

The Kubernetes community today announced the release of Kubernetes 1.10, just a few weeks since it graduated from CNCF incubation. As a founding member of the CNCF and the primary authors of Kubernetes, Google continues to be the largest contributor to the project in this release, as well as reviewer of contributions and mentor to community members. At Google we believe growing a vibrant community helps deliver a platform that's open and portable, so users benefit by being able to run their workloads consistently anywhere they want.

In this post, we highlight a few elements of the 1.10 release that we helped contribute to.

Container storage plugins


The Kubernetes implementation of the Container Storage Interface (CSI) has moved to beta in Kubernetes 1.10. CSI enables third-party storage providers to develop solutions outside of the core Kubernetes codebase. Because these plugins are decoupled from the core codebase, installing them is as easy as deploying a Pod to your cluster.

Saad Ali (chair of SIG-Storage) is a primary author of both the CSI specification and Kubernetes' implementation of the specification. "Kubernetes provides a powerful volume plugin system that makes it easy to consume different types of block and file storage,” he explains. “However, adding support for new volume plugins has been challenging. With the adoption of the Container Storage Interface, the Kubernetes volume layer is finally becoming truly extensible. Third-parties can now write and deploy plugins exposing new storage systems in Kubernetes without ever having to touch the core Kubernetes code. Ultimately this will give Kubernetes and Kubernetes Engine users more options for the storage that backs their stateful containerized workloads."

Custom resolver configuration


A key feature of Kubernetes is being able to refer to your Services by a simple DNS name, rather than deal with the complexities of an external discovery service. While this works great for internal names, some Kubernetes Engine customers reported that it caused an overload on the internal DNS server for workloads that primarily look up external names.

Zihong Zheng implemented a feature to allow you to customize the resolver on a per-pod basis. "Kubernetes users can now avoid this trade-off if they want to, so that neither ease of use nor flexibility are compromised," he says. Building this upstream means that the feature is available to Kubernetes users wherever they run.


Device plugins and GPU support


Also moving to beta in 1.10 are Device Plugins, an extension mechanism that lets device vendors advertise their resources to the kubelet without changing Kubernetes core code. A primary use case for device plugins is connecting GPUs to Kubernetes.

Jiaying Zhang is Google's feature lead for device plugins. She worked closely with device vendors to understand their needs, identify common requirements, come up with an execution plan, and work with the OSS community to build a production-ready system. Kubernetes Engine support for GPUs is built on the Device Plugins framework, and our early access customers influenced the feature as it moved to production readiness in Kubernetes 1.10.

API extensions


Googler Daniel Smith (co-chair of SIG API Machinery) first proposed the idea of API extension just a couple months after Kubernetes was open-sourced. We now have two methods for extending the Kubernetes API: Custom Resource Definitions (formerly Third-Party Resources), and API Aggregation, which moves to GA in Kubernetes 1.10. Aggregation, which is used to power ecosystem extensions like the Service Catalog and Metrics Server, allows independently built API server binaries to be hosted through the Kubernetes master, with the same authorization, authentication and security configurations on both. “We’ve been running the aggregation layer in every Google Kubernetes Engine cluster since 1.7 without difficulties, so it’s clearly time to promote this mechanism to GA,” says Daniel. "We’re working to provide a complete extensibility solution, which involves getting both CRDs and admission control webhooks to GA by the end of the year.”

Use Kubernetes to run your Spark workloads


Google's contributions to the open Kubernetes ecosystem extend farther than the Kubernetes project itself. Anirudh Ramanathan (chair of SIG-Big Data) led the upstream implementation of native Kubernetes support in Apache Spark 2.3, a headline feature in that release. Along with Yinan Li, we are hard at work on a Spark Operator, which lets you run Spark workloads in an idiomatic Kubernetes fashion.

Paired with the priority and preemption feature implemented by Bobby Salamat (chair of SIG-Scheduling) and David Oppenheimer (co-author of the Borg paper) you'll soon be able to increase the efficiency of your cluster by using Spark to schedule batch work to run only when the cluster has free resources.

Growing the community


We’re also heavily invested in mentoring for the Kubernetes project. Outreachy is an internship program that helps traditionally underrepresented groups learn and grow tech skills by contributing to open-source projects. Kubernetes' SIG-CLI participated in Outreachy over the 1.10 timeframe with Google's Antoine Pelisse as mentor. With his help, Yolande Amate from Cameroon and Ellen Korbes from Brazil took on the challenge of making improvements to the "kubectl create" and "kubectl set" commands.

With the internship over, Ellen is now a proud Kubernetes project member (and has written a series of blog posts about her path to contribution), and Yolande continues to submit PRs and is working toward her membership.


1.10 available soon on Kubernetes Engine


This latest version of Kubernetes will start rolling out to alpha clusters on Kubernetes Engine in early April. If you want to be among the first to access it on your production clusters, join our early access program today.

If you haven’t tried GCP and Kubernetes Engine before, you can quickly get started with our $300 free credits.

Getting to know Cloud Armor — defense at scale for internet-facing services

By Prajakta Joshi, Product Manager, Cloud Networking

We know that you have the tough job of quickly and responsively serving your users, while also simultaneously defending your internet-facing services from malicious attacks. That’s why we announced Cloud Armor, a new DDoS and application defense service, at the CEO Security Forum in New York last week. It’s based on the same technologies and global infrastructure that we use to protect Google services like Search, Gmail and YouTube.

Economy of scale


Our security experts work around the clock to defend Google’s core services from a wide variety of malicious attacks. Metrics from DDoS attacks targeting Google services over the past decade reveal attack volumes have increased exponentially across several axes: bits per second (bps), packets per second (pps) and HTTP(S) queries per second (qps).
click to enlarge

“Absorbing the largest attacks requires the bandwidth needed to watch half a million YouTube videos at the same time... in HD.”  
— Dr. Damian Menscher, DDoS Defense, Google
To defend against this threat, we deploy edge infrastructure and security systems to mitigate attacks targeting our services—and this same infrastructure underpins Google Cloud. With global HTTP(S) load balancing, the first Google Cloud Platform (GCP) service to support Cloud Armor, you get built-in defense against infrastructure DDoS attacks. No additional configuration, other than to configure load balancing, is required.

Defense is a collaborative effort.
“We work closely with several industry groups to track emerging threats, allowing us to both protect ourselves and others. In addition, we host krebsonsecurity.com and other frequent targets to ensure we are among the first to see new attack methods. This lets us design defenses and dismantle botnets before they have a chance to grow.”  
— Dr. Damian Menscher, DDoS Defense, Google
Sharing resources across Google and Google Cloud services allows us to easily absorb the largest attacks, and also ensure that an attack on one customer doesn’t affect others.

Cloud Armor: Policy driven application defense at scale


Cloud Armor works in conjunction with global HTTP(S) load balancing and enables you to deploy and customize defenses for your internet-facing applications. Similar to global HTTP(S) load balancing, Cloud Armor is delivered at the edge of Google’s network, helping to block attacks close to their source. It's built on three pillars: a policy framework, a rich rules language and global enforcement infrastructure.

click to enlarge

"Cloud Armor is a great example of how Google continues to innovate on its pervasive defense-in-depth security strategy, providing a rich layer of security control that can be managed at the network edge."  
— Matt Hite, Network Engineer, Evernote

Cloud Armor features and functionality


With Cloud Armor, you can:
  • defend your services against infrastructure DDoS attacks via HTTP(S) load balancing 
  • configure security policies, specify rules and order of evaluation for these rules 
  • allow, block, preview and log traffic 
  • deploy IP whitelists and blacklists for both IPv4 and IPv6 traffic 
  • create custom rules using a rich rules language to match traffic based on any combination of Layer 3 and HTTP(S) request parameters and allow or block this traffic (in alpha) 
  • enable geolocation-based control, and application-aware defense for SQL Injection (SQLi) and Cross-site Scripting (XSS) attacks (in alpha)
With the above foundation in place, we look forward to expanding Cloud Armor’s capabilities in the coming months.

Cloud Armor Security policy framework


Cloud Armor configuration is driven by security policies. To deploy Cloud Armor, you must create a security policy, add rules, and then attach this policy to one or more HTTP(S) load balancing backend services.

A Cloud Armor security policy is comprised of one or more rules, where each rule specifies the parameters to look for in the traffic, the action to take if the traffic matches these parameters, and a priority value that determines the position of this rule in the policy hierarchy.

click to enlarge


Cloud Armor allows you to create multiple policies per project. You can customize the defense for a subset of backend services by creating a policy specifically for these services.
click to enlarge
Below, we show how to configure IP Blacklists and whitelists using Cloud Armor:


Cloud Armor Rules Language (in alpha)


Cloud Armor rules language enables you to customize defenses for your specific requirements. Often attackers use multiple well-known and custom malicious patterns to attempt bringing your service down. Custom rules enable you to configure specific attack patterns to look for in the traffic and then block this traffic at scale.

Here’s an example of a custom rule to defend against an attack seen to be originating from US and containing a specific cookie and user-agent.

Configuration using gCloud CLI:


Configuration using console:
click to enlarge
For the most common application-aware attacks, Cloud Armor provides two pre-configured rules: Cross-site Scripting (‘xss-canary’) and SQL Injection (‘sqli-canary’) defenses. In the example below, we configure an SQL injection defense rule in policy “sql-injection-dev” using gCloud CLI:
Below, you can see the SQLi defense rule, along with other rules, in the policy:
You can request Alpha access to these features by signing up using this form.

Visibility into blocked and allowed traffic


You can view the allowed and blocked traffic in Stackdriver as shown below:

Partner ecosystem


We have a rich ecosystem of security providers who offer solutions that complement Cloud Armor’s capabilities. You can use these in conjunction with global HTTP(S) load balancing and Cloud Armor to build a comprehensive security solution. Learn more about our security partners here.

Get started today


Cloud Armor is for everyone deploying internet-facing services in Google Cloud. Learn more by visiting the Cloud Armor website. We look forward to your feedback!

Kubernetes Engine Private Clusters now available in beta

By Ines Envid and Manjot Pahwa, Product Managers, Google Cloud Networking

Google Cloud Platform (GCP) employs several security measures to help ensure authenticity, privacy and integrity of your data in transit. As enterprise users turn to Google Kubernetes Engine (GKE) as their preferred deployment model, they too require the same levels of privacy for their data.

Today, we're excited to announce the beta launch of Kubernetes Engine Private Clusters. Now, Kubernetes Engine allows you to deploy clusters privately as part of the Google Virtual Private Cloud (VPC), which provides an isolated boundary where you can securely deploy your applications. With Kubernetes Engine Private Clusters, your cluster’s nodes can only be accessed from within the trusted VPC. In addition, private clusters protect the master node from unwanted access, as the master is completely blocked from access from the internet by default.

In the Kubernetes Engine Private Cluster model, your nodes have access to the rest of your VPC private deployments, including private access to Google managed services such as gcr.io, Google Cloud Storage and Google BigQuery. Access to the internet isn’t possible unless you set up additional mechanisms such as a NAT gateway.

Kubernetes Engine Private Clusters greatly simplify PCI-DSS compliance of your deployments, by limiting how a cluster can be reached from outside of a private network.

Let's take a closer look at how Kubernetes Engine Private Clusters fit into GCP’s private VPC model.

Get started with Private Clusters on Kubernetes Engine


The following tutorial highlights how you can enable Private Clusters for your deployments. In this private cluster model, the Kubernetes Engine cluster nodes are allocated private IP addresses and the master is protected from internet access. As you can see in the example below, you enable a Kubernetes Engine Private Cluster at cluster creation time, selecting the private IP range within your RFC 1918 IP space to use for your master, nodes, pods and services.

Note that you must deploy Kubernetes Engine Private Clusters with IP Aliases enabled. It also requires cluster version 1.8.5 or later.

The following diagram displays the internals of private clusters:

The fastest way to get started is to use the UI during cluster creation:
Alternatively, you can also create your private cluster with the GCP gcloud CLI: 

# Create a Private Cluster with IP Alias auto-subnetwork)
gcloud beta container clusters create  --project=<project_id>>
--zone= --private-cluster --master-ipv4-cidr= 
--enable-ip-alias --create-subnetwork=""</master_cidr_block></zone></project_id></cluster></code>

The Master Authorized Network firewall protects access to the Kubernetes Engine master. When Kubernetes Engine Private Clusters is enabled, it's set to “default deny,” making your master inaccessible from the public internet at creation time.


Try it out today!

Create a Kubernetes Engine Private Cluster today. Stay tuned for more updates in this space as we continue to invest in Kubernetes Engine to ensure customers get defense-in-depth security features.

Interested in optimal load balancing?


Do you want to get access to a more container-native load balancing approach in Kubernetes Engine? Sign up here!

Take charge of your sensitive data with the Cloud Data Loss Prevention (DLP) API

By Scott Ellis, GCP Product Management

This week, we announced the general availability of the Cloud Data Loss Prevention (DLP) API, a Google Cloud security service that helps you discover, classify and redact sensitive data at rest and in real-time.

When it comes to properly handling sensitive data, the first step is knowing where it exists in your data workloads. This not only helps enterprises more tightly secure their data, it’s a fundamental component of reducing risk in today’s regulatory environment, where the mismanagement of sensitive information can come with real costs.

The DLP API is a flexible and robust tool that helps identify sensitive data like credit card numbers, social security numbers, names and other forms of personally identifiable information (PII). Once you know where this data lives, the service gives you the option to de-identify that data using techniques like redaction, masking and tokenization. These features help protect sensitive data while allowing you to still use it for important business functions like running analytics and customer support operations. On top of that, the DLP API is designed to plug into virtually any workload—whether in the cloud or on-prem—so that you can easily stream in data and take advantage of our inspection and de-identification capabilities.

In light of data privacy regulations like GDPR, it’s important to have tools that can help you uncover and secure personal data. The DLP API is also built to work with your sensitive workloads and is supported by Google Cloud’s security and compliance standards. For example, it’s a covered product under our Cloud HIPAA Business Associate Agreement (BAA), which means you can use it alongside our healthcare solutions to help secure PII.

To illustrate how easy it is to plug DLP into your workloads, we’re introducing a new tutorial that uses the DLP API and Cloud Functions to help you automate the classification of data that’s uploaded to Cloud Storage. This function uses DLP findings to determine what action to take on sensitive files, such as moving them to a restricted bucket to help prevent accidental exposure.

In short, the DPI API is a useful tool for managing sensitive data—and you can take it for a spin today for up to 1 GB at no charge. Now, let’s take a deeper look at its capabilities and features.

Identify sensitive data with flexible predefined and custom detectors

Backed by a variety of techniques including machine learning, pattern matching, mathematical checksums and context analysis, the DLP API provides over 70 predefined detectors (or “infotypes”) for sensitive data like PII and GCP service account credentials.

You can also define your own custom types using:
  • Dictionaries — find new types or augment the predefined infotypes 
  • Regex patterns — find your own patterns and define a default likelihood score 
  • Detection rules — enhance your custom dictionaries and regex patterns with rules that can boost or reduce the likelihood score based on nearby context or indicator hotwords like “banking,” “taxpayer,” and “passport.”

Stream data from virtually anywhere

Are you building a customer support chat app and want to make sure you don’t inadvertently collect sensitive data? Do you manage data that’s on-prem or stored on another cloud provider? The DLP API “content” mode allows you to stream data from virtually anywhere. This is a useful feature for working with large batches to classify or dynamically de-identify data in real-time. With content mode, you can scan data before it’s stored or displayed, and control what data is streamed to where.

Native discovery for Google Cloud storage products

The DLP API has native support for data classification in Cloud Storage, Cloud Datastore and BigQuery. Just point the API at your Cloud Storage bucket or BigQuery table, and we handle the rest. The API supports:
  • Periodic scans — trigger a scan job to run daily or weekly 
  • Notifications — launch jobs and receive Cloud Pub/Sub notifications when they finish; this is great for serverless workloads using Cloud Functions
  • Integration with Cloud Security Command CenterAlpha
  • SQL data analysis — write the results of your DLP scan into the BigQuery dataset of your choice, then use the power of SQL to analyze your findings. You can build custom reports in Google Data Studio or export the data to your preferred data visualization or analysis system.
A summary report of DLP findings on recent scans


Redact data from free text and structured data at the same time

With the DLP API, you can stream unstructured free text, use our powerful classification engine to find different sensitive elements and then redact them according to your needs. You can also stream in tabular text and redact it based on the record types or column names. Or do both at the same time, while keeping integrity and consistency across your data. For example, you can take a social security number that’s classified in a comment field as well as in a structured column, and it generates the same token or hash.

Extend beyond redaction with a full suite of de-identification tools


From simple redaction to more advanced format-preserving tokenization, the DLP API offers a variety of techniques to help you redact sensitive elements from your data while preserving its utility.

Below are a few supported techniques:
Transformation type
Description
Replacement
Replaces each input value with infoType name or a user customized value
Redaction
Redacts a value by removing it
Mask or partial mask
Masks a string either fully or partially by replacing a given number of characters with a specified fixed character
Pseudonymization
with cryptographic hash
Replaces input values with a string generated using a given data encryption key
Pseudonymization
with format preserving token
Replaces an input value with a “token,” or surrogate value, of the same length using format-preserving encryption (FPE) with the FFX mode of operation
Bucket values
Masks input values by replacing them with “buckets,” or ranges within which the input value falls
Extract time data
Extracts or preserves a portion of dates or timestamps

The Cloud DLP API can also handle standard bitmap images such as JPEGs and PNGs. Using optical character recognition (OCR) technology, the DLP API analyzes the text in images to return findings or generate a new image with the sensitive findings blocked out.

Measure re-identification risk with k-anonymity and l-diversity


Not all sensitive data is immediately obvious like a social security number or credit card number. Sometimes you have data where only certain values or combinations of values identify an individual, for example, a field containing information about an employee's job title doesn’t identify most employees. However, it does single out individuals with unique job titles like "CEO" where there’s only one employee with this title. Combined with other fields such as company, age or zip code, you may arrive at a single, identifiable individual. To help you better understand these kinds of quasi-identifiers, the DLP API provides a set of statistical risk analysis metrics. For example, risk metrics such as k-anonymity can help identify these outlier groups and give you valuable insights into how you might want to further de-identify your data, perhaps by removing rows and bucketing fields.

Use k-anonymity to help find identifiable individuals in your datasets


Integrate the DLP API into your workloads across the cloud ecosystem


The DLP API is built to be flexible and scalable, and includes several features to help you integrate it into your workloads, wherever they may be.

  • DLP templates — Templates allow you to configure and persist how you inspect your data and define how you want to transform it. You can then simply reference the template in your API calls and workloads, allowing you to easily update templates without having to redeploy new API calls or code.
  • Triggers — Triggers allow you to set up jobs to scan your data on a periodic basis, for example, daily, weekly or monthly. 
  • Actions — When a large scan job is done, you can configure the DLP API to send a notification with Cloud Pub/Sub. This is a great way to build a robust system that plays well within a serverless, event-driven ecosystem.

The DLP API can also integrate with our new Cloud Security Command Center Alpha, a security and data risk platform for Google Cloud Platform that helps enterprises gather data, identify threats, and act on them before they result in business damage or loss. Using the DLP API, you can find out which storage buckets contain sensitive and regulated data, help prevent unintended exposure, and ensure access is based on need-to-know. Click here to sign up for the Cloud Security Command CenterAlpha.
The DLP API integrates with Cloud Security Command Center to surface risks associated with sensitive data in GCP
Sensitive data is everywhere, but the DLP API can help make sure it doesn’t go anywhere it’s not supposed to. Watch this space for future blog posts that show you how to use the DLP API for specific use cases.

Expanding MongoDB Atlas availability on GCP

By Kent Smith, Cloud Customer Engineer

With over 35 million downloads and customers ranging from Cisco to Metlife to UPS, MongoDB is one of the most popular NoSQL databases for developers and enterprises alike.

MongoDB is available on Google Cloud Platform (GCP) through MongoDB’s simple-to-use, fully managed Database as a Service (DBaaS) product, MongoDB Atlas. With MongoDB Atlas, you get a globally distributed database with cross-region replication, multi-region fault tolerance and the ability to provide fast, responsive read access to data users around the globe. You can even configure your clusters to survive the outage of an entire cloud region.

Now, thanks to strong demand for MongoDB Atlas from GCP customers, Google and MongoDB have expanded the availability of MongoDB Atlas on GCP, making it available across most GCP regions as well as on Cloud Launcher.
With this expanded geographic availability, you can now join the wide variety of organizations around the world, from innovators in the social media space to industry leaders in energy, that are already running MongoDB on GCP.

How MongoDB Atlas works on GCP 


But first, to understand how MongoDB Atlas works on GCP, let’s discuss the architecture of a MongoDB cluster.

MongoDB maintains multiple copies of data called replica sets using native replication. A replica set is a fully self-healing unit that helps prevent database downtime. Failover is fully automated, eliminating the need for administrators to intervene manually.
You can configure the number of replicas in a MongoDB replica set; a larger number of replicas provides increased data availability and protection against database downtime (e.g., in case of multiple machine failures, rack failures, data center failures or network partitions). It's recommended that any MongoDB cluster include at least three replica set members, ideally geographically distributed to ensure no single point of failure.

Alternatively, you can configure operations to write to multiple replicas before returning the write acknowledgement to the application, for near-synchronous replication. This multi-data center awareness enables global data distribution and separation between operational and analytical workloads. Replica sets also provide operational flexibility by providing a way to upgrade hardware and software without taking the database offline.

MongoDB provides horizontal scale-out in the cloud via a process called sharding. Sharding distributes data across multiple physical partitions (shards). Sharding allows MongoDB to address the hardware limitations of a single server without adding complexity to the application. MongoDB automatically balances the data in the cluster as the data grows or the size of the cluster increases or decreases.

Sharding is transparent to applications; whether there's one or one hundred shards, the application code for querying MongoDB is the same.
MongoDB provides multiple sharding policies, letting you distribute data across a cluster according to query patterns or data locality.

MongoDB clusters managed by MongoDB Atlas on GCP are organized into Projects. All clusters within a Project live inside a single Virtual Private Cloud (VPC) per region, ensuring network isolation from other customers. By default, clusters contain three MongoDB replica set members, which are distributed across the available zones within a single GCP region.

Creating a globally distributed database with MongoDB Atlas


Configuring cross-region replication is easy from the MongoDB Atlas UI, as shown below:
Once configured, MongoDB Atlas automatically scales the storage on clusters and makes it easy to scale the compute (memory and CPU), to larger cluster sizes, or enable sharding in the UI or API, with no manual intervention.

We hope you take advantage of the availability of MongoDB Atlas across more GCP regions to reduce the operational overhead of setting up and scaling the database, letting you focus on building better apps, faster. We look forward to seeing what you build and hearing your feedback so we can continue to make GCP the best place for running MongoDB in the cloud.

More on MongoDB


Easy HPC clusters on GCP with Slurm

By Michael Basilyan, Product Manager; Wyatt Gorman and Keith Binder, HPC Specialists; and Annie Ma-Weaver, Partner Manager 

High performance and technical computing is all about scale and speed. Many applications, such as drug discovery and genomics, financial services and image processing require access to a large and diverse set of computing resources on demand. With more and faster computing power, you can convert an idea into a discovery, a hypothesis into a cure or an inspiration into a product. Google Cloud provides the HPC community with on-demand access to large amounts of high-performance resources with Compute Engine. But a challenge remains: how do you harness these powerful resources to execute your HPC jobs quickly, and seamlessly augment an existing HPC cluster with Compute Engine capacity?

To help with this problem, we teamed up with SchedMD to release a preview of tools that makes it easier to launch the Slurm workload manager on Compute Engine, and to expand your existing cluster when you need extra resources. This integration was built by the experts at SchedMD in accordance with Slurm best practices. Slurm is a leading open-source HPC workload manager used often in the TOP500 supercomputers around the world.
With this integration, you can easily launch an auto-scaled Slurm cluster on Compute Engine. The cluster auto-scales according to job requirements and queue depth. Once the Slurm cloud cluster is setup, you can also use Slurm to federate jobs from your on-premises cluster to the Slurm cluster running in Compute Engine. With your HPC cluster in the cloud, you can give each researcher, team or job a dedicated, tailor-fit set of elastic resources so they can focus on solving their problems rather than waiting in queues.
Let’s walk through launching a Slurm cluster on Compute Engine.

Step 1: Grab the Cloud Deployment Manager scripts from SchedMD’s Github repository. Review the included README.md for more information. You may want to customize the deployment manager scripts for your needs. Many cluster parameters can be configured in the included slurm-cluster.yaml file.

At a minimum, you need to edit slurm-cluster.yaml to paste in your munge_key and specify your GCP username in default_users and the Slurm version you want to use (e.g., 17.11.5).

Step 2: Run the following command from the Cloud Shell or your local terminal with gcloud command installed:

gcloud deployment-manager deployments create slurm --config 
slurm-cluster.yaml

Then, navigate to the Deployment Manager section of the developer console and observe that your deployment is successful.
Step 3: If you navigate to the Compute Engine section of the developer console, you’ll see that Deployment Manager created a number of VM instances for you, among them a Slurm login node. After the VMs are provisioned, Slurm will be installed and configured on the VMs. You can now SSH into the login node by clicking the SSH button in the console or by running gcloud compute ssh login1 --zone=us-west1-a (Note: You may need to change the zone if you modified it in the slurm-cluster.yaml file.

Once you’ve logged in, you can interact with Slurm and submit jobs as usual using sbatch. For example, copy the sample script below into a new file called slurm-sample1.sh:

#!/bin/bash
#
#SBATCH --job-name=hostname_sleep_sample
#SBATCH --output=out_%j.txt
#
#SBATCH --nodes=2

srun hostname
sleep 60

Then, submit it with:

sbatch slurm-sample1.sh

You can then observe the job being distributed to the compute nodes using the sinfo and squeue commands. Notice how if the submitted job requires more resources than initially deployed, new instances will be automatically created, up to the maximum specified in slurm-cluster.yaml. To try this, set #SBATCH --nodes=4 and resubmit the job. Once the ephemeral compute instances are idle for a period of time specified, they'll be deprovisioned.

Note that for your convenience the deployment manager scripts set up NFS as part of the deployment.

Check out the included README for more information and if you need help getting started with Slurm check out the quick start guide or contact SchedMD.

Building trust through Access Transparency

By Joseph Valente, Product Manager

Auditability ranks at the top of cloud adopters’ security requirements. According to an MIT Sloan Management Review survey of more than 500 IT and business executives, 87% of respondents cited auditability as an important factor in evaluating cloud security—second only to a provider’s ability to prevent data compromises. While Google’s Cloud Audit Logging and similar products help answer the question of which of your administrators did what, where, when and why on your cloud objects, you’ve traditionally lost this audit trail once support is engaged. This is why we’re pleased to introduce Access Transparency, a new logs product unique to Google Cloud Platform (GCP) that provides an audit trail of actions taken by Google Support and Engineering when they interact with your data and system configurations on Google Cloud.

Access Transparency logs are available in beta for Compute Engine, App Engine, Cloud Identity and Access Management, Cloud Key Management Service, Cloud Storage and Persistent Disks— with more services becoming available throughout the year. Together, Cloud Audit Logs and Access Transparency Logs provide a more comprehensive view of admin activity in your cloud deployment.

Expanding your visibility with Access Transparency 


In the limited situations that access by Google employees does occur, Access Transparency logs are generated in near-real time and delivered to your Stackdriver Logging console in the same manner as Cloud Audit Logs. The logs not only show what resources were accessed and the operations performed, they also show the justification for that action. For example, they may include the ticket number you filed with support asking for help.
click to enlarge

You can also choose to export your Access Transparency logs into BigQuery and Cloud Storage, or to other tools in your existing audit pipeline through Cloud Pub/Sub. This allows you to integrate with your existing audit pipeline, where you may already be exporting your Cloud Audit Logs. You can then audit your Access Transparency logs with a combination of automated and manual review, in the same way you would with audit logs of your own internal activity.

Enabled by industry-leading data protection controls 


At Google Cloud, our philosophy is that our customers own their data, and we do not access that data for any reason other than those necessary to fulfill our contractual obligations to you. Technical controls require valid business justifications for any access to your content by support or engineering personnel. These structured justifications are used to generate your Access Transparency logs. Google also performs regular audits of accesses by administrators as a check on the effectiveness of our controls.

This system is built around limiting what employees can do, with multi-step processes to minimize the likelihood of misjudgment, and transparency to allow review of actions. Feedback loops also exist between Google’s audits and customer feedback to continue improving our processes and further limit the need to access your data in order to solve your problems.

Getting started with Access Transparency 


Access Transparency is available at no additional charge to customers with Platinum or Gold Support coverage, however spaces in our beta are limited. To apply for access, use our signup form. To find out more about Access Transparency, read the Access Transparency Documentation, or contact your dedicated support representative.

Access Transparency also continues to be available through SAP’s Data Custodian solution, which uses Access Transparency and other logs to support a managed GRC solution for your GCP deployments. For more information on Data Custodian, visit the SAP website.

Introducing new ways to protect and control your GCP services and data

By Jennifer Lin, Director of Product Management, GCP Security and Privacy

They say security is a process, not a destination, and that certainly rang true as we prepared for today’s CEO Security Forum in New York, where we’re making more than 20 security announcements across the Google Cloud portfolio.

When it comes to Google Cloud Platform (GCP), our goal is to continuously improve on the strong foundation that we’ve built over the years, and help you build out a secure, scalable environment. Here’s an overview of our GCP-related news. Stay tuned over the coming days for deeper dives into some of these new products.

1. Keep sensitive data private with VPC Service Controls Alpha


If your organization is looking to take advantage of the fully managed GCP technologies for big data analytics, data processing and storage, but has hesitated to put sensitive data in the cloud outside your secured network, our new VPC Service Controls provide an additional layer of protection to help keep your data private.

Currently in alpha, VPC Service Controls create a security perimeter around data stored in API-based GCP services such as Google Cloud Storage, BigQuery and Bigtable. This helps mitigate data exfiltration risks stemming from stolen identities, IAM policy misconfigurations, malicious insiders and compromised virtual machines.

With this managed service, enterprises can configure private communication between cloud resources and hybrid VPC networks using Cloud VPN or Cloud Dedicated Interconnect. By expanding perimeter security from on-premise networks to data stored in GCP services, enterprises can feel confident about storing their data in the cloud and accessing it from an on-prem environment or cloud-based VMs.

VPC Service Controls also take security a step further with context-aware access control for your cloud resources using the Access Context Manager feature. Enterprises can create granular access control policies in Access Context Manager based on attributes like user location and IP address. These policies help ensure the appropriate security controls are in place when granting access to cloud resources from the internet.

Google Cloud is the first cloud provider to offer virtual security perimeters for API-based services with simplicity, speed and flexibility that far exceed what most organizations can achieve in a physical, on-premises environment.

Get started by signing up for the upcoming VPC Service Controls beta.

2. Get insight into data and application risk with Cloud Security Command Center Alpha


As organizations entrust more applications to the cloud, it can be tough to understand the extent of your cloud assets and the risks against them. The new Cloud Security Command Center (Cloud SCC), currently in alpha, lets you view and monitor an inventory of your cloud assets, scan storage systems for sensitive data, detect common web vulnerabilities and review access rights to your critical resources—all from a single, centralized dashboard.

Cloud SCC provides deep views into the security status and health of GCP services such as App Engine, Compute Engine, Cloud Storage, and Cloud Datastore. It integrates with the DLP API to help identify sensitive information, and with Google Cloud Security Scanner to uncover vulnerabilities such as cross-site-scripting (XSS) and Flash injection. Use it to manage access control policies, receive alerts about unexpected changes through an integration with Forseti, the open-source GCP security toolkit, and detect threats and suspicious activity with Google anomaly detection as well as security partners such as Cloudflare, CrowdStrike, Dome9, Palo Alto Networks, Qualys and RedLock.

Get started by signing up for the Cloud SCC alpha program.

3. Expand Your visibility with Access Transparency


Trust is paramount when choosing a cloud provider. We want to be as open and transparent as possible, allowing customers to see what happens to their data. Now, with Access Transparency, we’ll provide you with an audit log of authorized administrative accesses by Google Support and Engineering, as well as justifications for those accesses, for many GCP services, and we’ll be adding more throughout the year. With Access Transparency, we can continue to maintain high performance and reliability for your environment while remaining accountable to the trust you place in our service.

Access Transparency logs are generated in near-real time, and appear in your Stackdriver Logging console in the same way that your Cloud Audit Logs do, with the same protections you would expect from audit-grade logs. You can export Access Transparency logs into BigQuery or Cloud Storage for storage and archiving, or via PubSub into your existing audit pipeline or SIEM tooling for further investigation and review.

The combination of Google Cloud Audit Logs and Access Transparency logs gives you a more comprehensive view into administrative activity in your GCP environment.

To learn more about Access Transparency, visit the product page, where you can find out more and sign up for the beta program.

4. Avoid Denial of Service with Cloud Armor


If you run internet-facing services or apps, you have a tough job: quickly and responsively serving traffic to your end users, while simultaneously protecting against malicious attacks trying to take your services down. Here at Google, we’re no stranger to this phenomenon, and so today, we’re announcing Cloud Armor, a Distributed Denial of Service (DDoS) and application defense service that’s based on the same technologies and global infrastructure that we use to protect services like Search, Gmail and YouTube.

Global HTTP(S) Load Balancing provides built-in defense against Infrastructure DDoS attacks. No additional configuration, other than to configure load balancing, is required to activate this DDoS defense. Cloud Armor works with Cloud HTTP(S) Load Balancing, provides IPv4 and IPv6 whitelisting/blacklisting, defends against application-aware attacks such as cross-site scripting (XSS) and SQL injection (SQLi), and delivers geography-based access control.

A sophisticated rules language and global enforcement engine underpin Cloud Armor, enabling you to create custom defenses, with any combination of Layer 3 to Layer 7 parameters, against multivector attacks (combination of two or more attack types). Cloud Armor gives you visibility into your blocked and allowed traffic by sending information to Stackdriver Logging about each incoming request and the action taken on that request by the Cloud Armor rule.

Learn more by visiting the Cloud Armor product page.

5. Discover, classify and redact sensitive data with the DLP API


Sensitive information is a fact of life. The question is—how do you identify it and help ensure it’s protected? Enter the Cloud Data Loss Prevention (DLP) API, a managed service that lets you discover, classify and redact sensitive information stored in your organization’s digital assets.

First announced last year, the DLP API is now generally available. And because the DLP API is, well, an API, you can use it on virtually any data source or business application, whether it’s on GCP services like Cloud Storage or BigQuery, a third-party cloud, or in your on-premises data center. Furthermore, you can use the DLP API to detect (and, just as importantly, redact) sensitive information in real-time, as well as in batch-mode against static datasets.

Our goal is to make the DLP API an extensible part of your security arsenal. Since it was first announced, we’ve added several new detectors, including one to identify service account credentials, as well as the ability to build your own detectors based on custom dictionaries, patterns and context rules.

Learn more by visiting the DLP API product page.

6. Provide simple, secure access for any user to cloud applications from any device with Cloud Identity


Last summer we announced Cloud Identity, a built-in service that allows organizations to easily manage users and groups who need access to GCP resources. Our new, standalone Cloud Identity product is a full Identity as a Service (IDaaS) solution that adds premium features such as enterprise security, application management and device management to provide enterprises with simple, secure access for any user to cloud applications from any device.

Cloud Identity enables and accelerates the use of cloud-centric applications and services, while offering capabilities to meet customer organizations where they are with their on-premise IAM systems and apps.

Learn more by visiting the Cloud Identity product page.

7. Extending the benefits of GCP security to U.S. federal, state and local government customers through FedRamp authorization


While Google Cloud goes to great lengths to document the security capabilities of our infrastructure and platforms, third-party validation always helps. We’re pleased to announce that GCP, and Google’s underlying common infrastructure, have received the FedRAMP Rev. 4 Provisional Authorization to Operate (P-ATO) at the Moderate Impact level from the FedRAMP Joint Authorization Board (JAB). GCP’s certification encompasses data centers in many countries, so customers can take advantage of this certification from multiple Google Cloud regions.

Agencies and federal contractors can request access to our FedRAMP package by submitting a FedRAMP Package Access Request Form.

8. Take advantage of new security partnerships

In addition to today’s security announcements, we’ve also been working with several security companies to offer additional solutions that complement GCP’s capabilities. You can read about these partnerships in more detail in our partnerships blog post.

Today we’re announcing new GCP partner solutions, including:
  • Dome9 has developed a compliance test suite for the Payment Card Industry Data Security Standard (PCI DSS) in the Dome9 Compliance Engine. 
  • Rackspace Managed Security provides businesses with fully managed security on top of GCP. 
  • RedLock’s Cloud 360 Platform is a cloud threat defense security and compliance solution that provides additional visibility and control for Google Cloud environments.
As always, we’re thrilled to be able to share with you the fruits of our experience running and protecting some of the world’s most popular web applications. And we’re honored that you’ve chosen to make GCP your home in the cloud. For more on today’s security announcements read our posts on the Google Cloud blog, the G Suite blog and the Connected Workspaces blog.

Expanding our Google Cloud security partnerships

By Vineet Bhan, Head of Security Partnerships and Ashish Verma, Technical Program Manager

As we discussed in today’s blog post, security is top of mind for many businesses as they move to the cloud. To help more businesses take advantage of the cloud’s security benefits, we’re working with several leading security providers to offer solutions that complement Google Cloud Platform’s capabilities, and enable customers to leverage their existing tools from these vendors in the cloud. These partner solutions cover a broad set of enterprise security needs, such as advanced threat prevention, compliance, container security, managed security services and more.

Today, we’re announcing new partnerships, new solutions by existing partners and new partner integrations in our Cloud Security Command Center (Cloud SCC), currently in alpha. Here’s a little more on what each of these partnerships will offer:

Auth0 offers the ability to secure cloud endpoints and seamlessly implement secure identity management into customer products and services. Whether the goal is to add additional authentication sources like social login, migrate users without requiring password resets or add multi-factor authentication, Auth0 provides a range of services to accomplish many identity-related tasks. Auth0’s platform supports multiple use cases (B2B, B2C, B2E, IoT, API) and integrates into existing tech stacks.

Check Point  can now secure multiple VPCs using a single CloudGuard security gateway to protect customer applications. A single CloudGuard gateway can monitor traffic in and out of more than one VPC network at any given time, providing a more efficient and scalable solution for running and securing workloads.

Cloudflare Web Application Firewall helps prevents attackers from compromising sensitive customer data, and helps protect customers from common vulnerabilities like SQL injection attacks, cross-site scripting and cross-site forgery. Additionally, integration with the Cloud Security Command CenterAlpha combines their intelligence with Google security and data risk insights to give customers a holistic view of their security posture.

Dome9 has developed a compliance test suite for the Payment Card Industry Data Security Standard (PCI DSS) in the Dome9 Compliance Engine. Using the Compliance Engine, Google Cloud customers can assess the compliance posture of their projects, identify risks and gaps, fix issues such as overly permissive firewall rules, enforce compliance requirements and demonstrate compliance in audits. The integration between the Dome9 Arc platform and the Cloud Security Command Center allows customers to consume and explore the results of assessments of the Dome9 Compliance Engine directly from Cloud SCC.

Fortinet provides scalable network protection for workloads in Google Cloud Platform (GCP). Its FortiGate provides next-generation firewall and advanced security, and its Fortinet Security Fabric integration enables single pane-of-glass visibility and policy across on-premises workloads and GCP for consistent hybrid cloud security.

Palo Alto Networks VM-Series Next Generation Firewall helps customers to securely migrate their applications and data to GCP, protecting them through application whitelisting and threat prevention policies. Native automation features allow developers and cloud architects to create “touchless” deployments and policy updates, effectively embedding the VM-Series into the application development workflow. The VM-Series on GCP can be configured to forward threat prevention, URL Filtering and WildFire logs of high severity to the Cloud Security Command Center to provide a consolidated view of a customer’s GCP security posture.

Qualys provides vulnerability assessments for Google Compute Engine instances. Users can get their vulnerability posture at a glance and drill down for details and actionable intelligence for the vulnerabilities identified. Customers can get this visibility within the Cloud Security Command Center by deploying the lightweight Qualys agents on the instances, baking them into images or deploying them directly into Compute Engine instances.

Rackspace Managed Security and Compliance Assistance provides additional active security on GCP to detect and respond to advanced cyber threats. Rackspace utilizes pre-approved actions to promptly remediate security incidents. It also complements the strategic planning, architectural guidance and 24x7x365 operational support available through Managed Services for GCP.

RedLock Cloud 360 Platform is a cloud threat defense security and compliance solution that provides additional visibility and control for GCP. RedLock collects and correlates disparate data sets from Google Cloud to determine the risk posture of a customer’s environment, then employs risk scoring algorithms to help prioritize and remediate the highest risks. Redlock’s integration with the Cloud Security Command Center provides customers with centralized visibility into security and compliance risks. As part of the integration, RedLock periodically scans a customer's Google Cloud environments and sends results pertaining to resource misconfigurations, compliance violations, network security risks and anomalous user activities.

StackRox augments Google Kubernetes Engine’s built-in security functions with a deep focus on securing the container runtime environment. StackRox’s core capabilities and functionality include network discovery and visualization of the application, detection of adversarial actions and detection of new attacks via machine-learning capabilities.

Sumo Logic Machine Data Analytics Platform offers enterprise-class monitoring, troubleshooting and security for mission-critical cloud applications. Sumo Logic platform integrates directly with GCP services through Google Stackdriver to collect audit and operational data in real-time so that customers can monitor and troubleshoot Google VPC, Cloud IAM, Cloud Audit, App Engine, Compute Engine, Cloud SQL, BigQuery, Cloud Storage, Kubernetes Engine and Cloud Functions, with more coming soon.

To learn more about our partner program, or to find a partner, visit our partner page.