• Spotify has migrated its production monitoring system, Heroic, from storing time series in Apache Cassandra to Cloud Bigtable and is writing over 360K data points per second.
• FIS is working on a bid for the SEC Consolidated Audit Trail (CAT) project, and was able to achieve 34 million reads/sec and 23 million writes/sec on Cloud Bigtable as part of its market data processing pipeline.
• Energyworx is building an IoT solution for the energy industry on Google Cloud Platform, using Cloud Bigtable to store smart meter data. This allows it to scale without building a large DevOps team to manage its storage backend.

Cloud SQL Second Generation vs. the competition

• Test: We used sysbench to simulate the same workload on three different services: Cloud SQL Second Generation, Amazon RDS for MySQL and Amazon Aurora.
• Result: Cloud SQL Second Generation outperformed RDS for MySQL and performed better than Aurora when active thread count is low, as is typical for many web applications.

Cloud SQL sustains higher TPS (transactions per second) per thread than RDS for MySQL. It outperforms Aurora in configurations of up to 16 threads.

Ubuntu setup
sudo apt-get update
sudo apt-get install \
  git automake autoconf libtool make gcc \
  Libmysqlclient-dev mysql-client-5.6


git clone https://github.com/akopytov/sysbench.git
(apply patch)
./autogen.sh 
./configure
make -j8


Test variables
export test_system=<test name>
export mysql_host=<mysql host>
export mysql_user=<mysql user>
export mysql_password=<mysql password>
export test_path=~/oltp_${test_system}_1
export test_name=01_baseline


Prepare test data
sysbench/sysbench \
  --mysql-host=${mysql_host} \
  --mysql-user=${mysql_user} \
  --mysql-password=${mysql_password} \
  --mysql-db="sbtest" \
  --test=sysbench/tests/db/parallel_prepare.lua \
  --oltp_tables_count=100 \
  --oltp-table-size=20000000 \
  --rand-init=on \
  --num-threads=16 \
  run


Run the benchmark:


mkdir -p ${test_path}
for threads in 1 2 4 8 16 32 64 128 256 512 1024
do 


sysbench/sysbench \
  --mysql-host=${mysql_host} \
  --mysql-user=${mysql_user} \
  --mysql-password=${mysql_password} \
  --mysql-db="sbtest" \
  --db-ps-mode=disable \
  --rand-init=on \
  --test=sysbench/tests/db/oltp.lua \
  --oltp-read-only=off \
  --oltp_tables_count=100 \
  --oltp-table-size=20000000 \
  --oltp-dist-type=uniform \
  --percentile=99 \
  --report-interval=1 \
  --max-requests=0 \
  --max-time=1800 \
  --num-threads=${threads} \
  run


Format the results:


Capture results in CSV format
grep "^\[" ${test_path}/${test_name}_*.out \
  | cut -d] -f2 \
  | sed -e 's/[a-z ]*://g' -e 's/ms//' -e 's/(99%)//' -e 's/[ ]//g' \
  > ${test_path}/${test_name}_all.csv


Plot the results in R
status <- NULL # or e.g. "[DRAFT]"
config <- "Amazon RDS (MySQL Multi-AZ, Aurora) vs. Google Cloud SQL Second Generation\nsysbench 0.5, 100 x 20M rows (2B rows total), 30 minutes per step"
steps <- c(1, 2, 4, 8, 16, 32, 64, 128, 256, 512)
time_per_step <- 1800
output_path <- "~/oltp_results/"
test_name <- "01_baseline"


results <- data.frame(
  stringsAsFactors = FALSE,
  row.names = c(
    "amazon_rds_multi_az",
    "amazon_rds_aurora",
    "google_cloud_sql"
  ),
  file = c(
    "~/amazon_rds_multi_az_1/01_baseline_all.csv",
    "~/amazon_rds_aurora_1/01_baseline_all.csv",
    "~/google_cloud_sql_1/01_baseline_all.csv"
  ),
  name = c(
    "Amazon RDS MySQL Multi-AZ",
    "Amazon RDS Aurora",
    "Google Cloud SQL 2nd Gen."
  ),
  color = c(
    "darkgreen",
    "red",
    "blue"
  )
)


results$data <- lapply(results$file, read.csv, header=FALSE, sep=",", col.names=c("threads", "tps", "reads", "writes", "latency", "errors", "reconnects"))


# TPS
pdf(paste(output_path, test_name, "_tps.pdf", sep=""), width=12, height=8)
plot(0, 0,
  pch=".", col="white", xaxt="n", ylim=c(0,2000), xlim=c(0,length(steps)),
  main=paste(status, "Transaction Rate by Concurrent Sysbench Threads", status, "\n\n"),
  xlab="Concurrent Sysbench Threads",
  ylab="Transaction Rate (tps)"
)
for(result in rownames(results)) {
  tps <- as.data.frame(results[result,]$data)$tps
  points(1:length(tps) / time_per_step, tps, pch=".", col=results[result,]$color, xaxt="n", new=FALSE)
}
title(main=paste("\n\n", config, sep=""), font.main=3, cex.main=0.7)
axis(1, 0:(length(steps)-1), steps)
legend("topleft", results$name, bg="white", col=results$color, pch=15, horiz=FALSE)
dev.off()


# Latency
pdf(paste(output_path, test_name, "_latency.pdf", sep=""), width=12, height=8)
plot(0, 0,
  pch=".", col="white", xaxt="n", ylim=c(0,2000), xlim=c(0,length(steps)),
  main=paste(status, "Latency by Concurrent Sysbench Threads", status, "\n\n"),
  xlab="Concurrent Sysbench Threads",
  ylab="Latency (ms)"
)
for(result in rownames(results)) {
  latency <- as.data.frame(results[result,]$data)$latency
  points(1:length(latency) / time_per_step, latency, pch=".", col=results[result,]$color, xaxt="n", new=FALSE)
}
title(main=paste("\n\n", config, sep=""), font.main=3, cex.main=0.7)
axis(1, 0:(length(steps)-1), steps)
legend("topleft", results$name, bg="white", col=results$color, pch=15, horiz=FALSE)
dev.off()


# TPS per Thread
pdf(paste(output_path, test_name, "_tps_per_thread.pdf", sep=""), width=12, height=8)
plot(0, 0,
  pch=".", col="white", xaxt="n", ylim=c(0,60), xlim=c(0,length(steps)),
  main=paste(status, "Transaction Rate per Thread by Concurrent Sysbench Threads", status, "\n\n"),
  xlab="Concurrent Sysbench Threads",
  ylab="Transactions per thread (tps/thread)"
)
for(result in rownames(results)) {
  tps <- as.data.frame(results[result,]$data)$tps
  threads <- as.data.frame(results[result,]$data)$threads
  points(1:length(tps) / time_per_step, tps / threads, pch=".", col=results[result,]$color, xaxt="n", new=FALSE)
}
title(main=paste("\n\n", config, sep=""), font.main=3, cex.main=0.7)
axis(1, 0:(length(steps)-1), steps)
legend("topleft", results$name, bg="white", col=results$color, pch=15, horiz=FALSE)
dev.off()

Cloud SQL Second Generation features

• Flexible backups: Schedule automatic daily backups or run them on-demand. Backups are designed not to affect performance.
• Precise recovery: Recover your instance to a specific point in time using point-in-time recovery.
• Easy clones: Clone your instance so you can test changes on a copy before introducing them to your production environment. Clones are exact copies of your databases, but they're completely independent from the source. Cloud SQL offers a streamlined cloning workflow.
• Automatic storage increase: Enable automatic storage increase and Cloud SQL will add storage capacity whenever you approach your limit.

• Open standards: We embrace the MySQL wire protocol, the standard connection protocol for MySQL databases, so you can access your database from nearly any application, running anywhere.
• Secure connections: Our new Cloud SQL Proxy creates a local socket and uses OAuth to help establish a secure connection with your application or MySQL tool. This makes secure connections easier for both dynamic and static IP addresses. For dynamic IP addresses, such as a developer’s laptop, you can help secure connectivity using service accounts, rather than modifying your firewall settings. For static IP addresses, you no longer have to set up SSL.

As a SaaS Company, we manage hundreds of instances for our customers. Cloud SQL is a major component of our stack and when we beta tested Cloud SQL, we were able to see fantastic performance over our large volume customers. We immediately migrated a few of our major customers as we saw 7x performance improvements of their queries.                                                                                     – Rajesh Manickadas, Director of Engineering, Orangescape 

As a mobile application company, data management is essential to delivering the best product for our clients. Google Cloud SQL enables us to manage databases that grow at rates such as 120 - 150 million data points every month. In fact, for one of our clients, a $6B Telecommunications Provider, their database adds ~15 GB of data every month. At peak time, we hit around 400 write operations/second and yet our API calls average return time is still under 73ms.                                                                                                                                                   – Andrea Michaud, Head of Client Services, www.TeamTracking.us

Next steps

Tableau customers continue to adopt Cloud SQL at a growing rate as they experience the benefits of rapid fire analytics in the cloud. With the significant performance improvements in Cloud SQL Second Generation, it’s likely that that adoption will grow even faster.                                                                            – Dan Kogan, Director of Product Marketing & Technology Partners, Tableau 

Looker is excited to support a Tier 1 integration for the Google's Cloud SQL Second Generation as it goes into General Availability. When you combine the Looker Data Platform's in-database analytics approach with Cloud SQL's fully-managed database offering, customers get a real-time analytics and visualization environment in the cloud, enabling anyone in the organization to make data-driven decisions.                                                                                                                                           – Keenan Rice, VP Strategic Alliances, Looker 

Migrating database applications to the cloud is a priority for many customers and we facilitate that process with Attunity Replicate by simplifying migrations to Google Cloud SQL while enabling zero downtime. Cloud SQL Second Generation delivers even better performance, reliability and security which are key for expanding deployments for enterprise customers. Customers can benefit from these enhanced abilities and we look forward to working with them helping to remove any data transfer hurdles.                                                                                                  – Itamar Ankorion, Chief Marketing Officer, Attunity 

Cloud SQL, Cloud Bigtable and Cloud Datastore are now generally available

Improved performance, security and platform support for databases

• Microsoft SQL Server images available on Google Compute Engine - Our top enterprise customers emphasize the importance of continuity for their mission-critical applications. The unique strengths of Google Compute Engine make it the best environment to run Microsoft SQL Server featuring images with built-in licenses (in beta), as well as the ability to bring your existing application licenses. Stay tuned for a post covering the details of running SQL Server and other key Windows workloads on Google Cloud Platform.
• Increased IOPS for Persistent Disk volumes - Database workloads are dependent on great block storage performance, so we're increasing the maximum read and write IOPS for SSD-backed Persistent Disk volumes from 15,000 to 25,000 at no additional cost, servicing the needs of the most demanding databases. This continues Google’s history of delivering greater price-performance over time with no action on the part of our customers.
• Custom encryption for Google Cloud Storage - When you need to store your database backups, you now have the added option of using customer-supplied encryption keys (CSEK). This feature allows Cloud Storage to be a zero-knowledge system without access to the keys and is now generally available.
• Low-latency for Google Cloud Storage Nearline storage - If you want a cost-effective way to store your database backups, Google Cloud Storage Nearline offers object storage at costs less than tape. Prior to today, retrieving data from Nearline incurred 3 to 5 seconds of latency per object. We've been continuously improving Nearline performance, and now it enables access times and throughput similar to Standard class objects. These faster access times and throughput give customers the ability to leverage big data tools such as Google BigQuery to run federated queries across your stored data.

from gcloud import storage


client = storage.Client(‘<your-project-id>’)
bucket = client.get_bucket('<your-bucket-name>')
blob = bucket.blob('my-test-file.txt')
blob.upload_from_string('this is test content!')

• Add just a single flag (--preemptible) in the gcloud compute instances create command, or by using one of our libraries
• Check a single box in the Developer Console create instance page
• Launch a quick and easy-to-use Spark/Hadoop cluster with Cloud Dataproc
• Autoscale Preemptible VMs with managed instance groups
• Render a movie with Zync and choose Preemptible VMs, which is now also up to 15% cheaper!

• Resources for Preemptible VMs come out of excess Google Cloud Platform capacity. The load on our Cloud Platform data centers varies with location and time of day, but is generally lowest on nights and weekends — the best time to run large Preemptible VM clusters.
• We avoid preempting too many VMs from a single customer and, given the choice, preempt VMs that were launched most recently. This might be a bit frustrating at first, but in the long run, this strategy helps minimize lost work across your cluster. And because we don’t bill for VMs preempted in the first 10 minutes, it saves on costs too.
• It’s a good idea to retry once or twice, even if you’ve been preempted early. Combining regular and Preemptible VMs in your clusters will ensure that tasks proceed at an adequate pace.
• Manage shutdown and preemption notices with a shutdown script that saves a job’s progress so that it can pick up where it left off, rather than start over from scratch.