By Pawel Siarkiewicz, Product Manager

A key feature of Google Compute Engine is managed instance groups, which allows you to manage collections of identical instances as a unit, to quickly deploy new VMs and ensure they're consistently configured. Today, we're pleased to announce a new managed instance group updater, to help you update your Compute Engine VMs programmatically and at scale. The updater is in beta and fully integrated into managed instance groups, making it easier than ever to update the software on your instances, patch and update them, and roll out staged and test deployments.

Specifically, the new managed instance group updater allows you to:

• Programmatically update your instances from one instance template to another 
• Specify how many instances to update at a time: one, several or all 
• Deploy additional instances (temporarily) to maintain capacity while old instances are being replaced 
• Restart, or recreate, all instances in the managed instance group without changing the template 
• Control the rate of deployment by configuring a pause between instance updates 

At first glance, all these options can be a bit overwhelming. Instead of explaining them one by one, let’s explore three typical use cases:

• Deploying a simple update 
• Deploying a canary test update 
• Recreating all instances 

We’ll show you how to use the managed instance group updater from the UI and gcloud command line, but you can also use it through the API.

Simple update 

Let’s start with the basics and deploy an update to the entire managed instance group, one instance at a time. The instance group my-instance-group starts with every instance running the template myapp-version-a, and we want to deploy myapp-version-b because we're deploying a new version of our software (we could also be doing so to patch/update the underlying OS).

gcloud beta compute instance-groups managed rolling-action 
start-update my-instance-group --version template=myapp-version-b

The managed instance group deploys myapp-version-b by first deleting an instance with myapp-version-a, while simultaneously creating an instance with myapp-version-b, waiting for that new instance to be healthy and then proceeding to the next instance until all of the instances are updated. If you want to roll back the update, just run the same command, but this time specify myapp-version-a as the target template.

gcloud beta compute instance-groups managed rolling-action 
start-update my-instance-group --version template=myapp-version-a

This is the default behavior for the updater, and it works really well for many typical deployments. If your managed instance group is large, however, updating one instance at a time might take too long. Let’s try this again, but this time, let’s update a quarter of the instances at a time.

gcloud beta compute instance-groups managed rolling-action 
start-update my-instance-group --version template=myapp-version-b 
--max-unavailable 25%

Note the new parameter, max-unavailable. It tells the updater how many instances can be taken offline at the same time, to be updated to the new template. Whether you have tens, hundreds or thousands of instances, the update proceeds at the rate you specify. If you’re wondering whether you can update all instances at the same time, the answer is yes. Just set max-unavailable to 100% and the managed instance group updates everything at once. This is a great option when consistent uptime doesn’t matter, for example in a testing environment or for batch computations.

Canary testing 

Now it’s time to consider a more advanced use case. Deploying new software to a subset of instances before committing fully is known as canary testing and is a best practice for managing robust, highly available systems. It allows you to test new software in a real environment while minimizing the impact of any issues.

For our example, let’s deploy myapp-version-b to just one instance.

gcloud beta compute instance-groups managed rolling-action 
start-update my-instance-group --version template=myapp-version-a 
--canary-version template=myapp-version-b,target-size=1

Here, the managed instance group deletes one instance of myapp-version-a and creates one instance of myapp-version-b. The update does not proceed further. If the instance group needs to scale up or down, it retains that one instance of myapp-version-b so that you can test it and make sure that it works as intended. When you’re ready, you can deploy it to a larger group, for example one half of the instances.

gcloud beta compute instance-groups managed rolling-action 
start-update my-instance-group --version template=myapp-version-a 
--canary-version template=myapp-version-b,target-size=50%

The managed instance group replaces instances running myapp-version-a with instances running myapp-version-b until the balance is 50/50. If autoscaling adds or removes instances, the managed instance group maintains this ratio, keeping the canary deployment at the level you set.

When you’re ready to commit to your new deployment, simply tell the updater that you want all of your instances to be running myapp-version-b. This is identical to what we previously did in the simple update.

gcloud beta compute instance-groups managed rolling-action 
start-update my-instance-group --version template=myapp-version-b

Recreating all instances 

Sometimes you don’t want to apply a new template, you just want to replace all of your instances, but keep the current template. This is a common pattern if you're using image families and want to make sure that your instances are using the latest version of an image. Starting a rolling replace of all instances follows the same pattern as starting an update.

gcloud beta compute instance-groups managed rolling-action replace 
my-instance-group --max-unavailable=10%

Notice that the replace action uses the same controls as max-unavailable that our earlier simple update example used to control how many instances to take offline at the same time.

Next steps 

We think these are the most common use cases for the managed instance groups updater, but realistically, we've just scratched the surface of everything that’s possible with it. Thanks to flexible controls like max-unavailable and max-surge, support for canary updates and different actuation commands such as start-update, restart and recreate, you now have a lot of options for maintaining your managed instance groups fleet. For more information, please see the documentation, and feel free to reach out to us with questions and feedback in the GCE discussion group.

You can try out the new managed instance group updater today in the console. If you’re new to Google Cloud Platform (GCP) you can also sign up to automatically get $300 worth of GCP credit to try out Compute Engine and the managed instance group updater.

By Tom Feiner and Eran Davidovich, System Operation Engineers, Waze

“At an abstract level, a deployment pipeline is an automated manifestation of your process for getting software from version control into the hands of your users.” ― Jez Humble, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation 

At Waze, we’ve been using Spinnaker for simultaneous deployments to multiple clouds since it was open sourced by Netflix in 2015.

However, implementing deployment pipelines can raise issues when your organization has more than 100 microservices (and growing), multiplied by the number of environments/regions/cloud providers.

• Managing hundreds of clusters and their deployment pipelines can quickly become a major maintenance burden. Once you have a few hundred deployment pipelines or more, keeping them up to date, or making a big change to all of them, is far from trivial. 
• There’s no code reuse among pipelines, even though many of them are probably identical. For example, bake stages, deploy stages and resize/clone/destroy stages tend to be identical except for some parameters. 
• Providing a paved road for deployments is easier said than done. Even though each development team should decide on their deployment strategy, it’s important to have a paved road — a way that works for most teams, which saves them time and effort. Start with that, and change as needed. However, there’s no easy way to create such a paved road and maintain it over time across all relevant pipelines in the system. 

Thankfully, Netflix — with contributions from the open-source community, including Google —added pipeline template support to Spinnaker, which solves these pain points and paves the way for multi-cloud infrastructure as code. With some much-appreciated help from the Google Spinnaker team, Waze is now using managed pipeline templates in production, and we plan to use them across our entire production system.

Unlike cloud vendor-specific solutions, Spinnaker managed pipeline templates work on top of all supported providers. (Currently: Azure, GCP, AWS, Kubernetes, OpenStack, App Engine and more are on the way.)

This means that for the first time, we can start realizing the dream of infrastructure as code across multiple clouds, whether you’re deploying to multiple public clouds, to a local Kubernetes on prem or to a mixed environment. We can see a future where automation, or some form of AI, decides to change instance types and other factors to reduce cost, improve utilization or mitigate real-time production incidents.

Runnable pipelines are composed of a pipeline template combined with pipeline configurations (using variables). Multiple configurations can use the same pipeline template as their base. In addition, pipeline templates support inheritance.

The benefits of pipeline templates 

Multi-cloud/provider continuous delivery and infrastructure as code (without vendor lock-in) 

Saving pipeline templates and configs in version control, subjecting them to code review and leaving an audit trail of why each infrastructure change was made are all extremely important for keeping your production system clean. These practices allow each production change to be understood, tracked in case there are any problems, and reproducible.

Being able to reproduce infrastructure enhances existing standards like reproducible builds, provides stability across the system and makes debugging, when required, easier. For example, changing an instance type, or adding a load balancer, is no longer just a “Clone” operation in Spinnaker; it’s now a code commit, a code review and a Jenkins job picking up the pipeline-template config change and publishing it to Spinnaker. After that, you just run the resulting pipeline(s) to apply the infrastructure change. You can easily track the infrastructure’s change history.

All of this is done in a way that supports multiple cloud providers. This may even be relevant for companies using Kubernetes on a single cloud provider, because they're required to interact with two sets of APIs and two control planes to manage their infrastructure. This is also why Kubernetes is treated as a separate cloud provider in Spinnaker. Spinnaker neatly abstracts all of that, while other solutions could lead to vendor lock-in.

Code reuse for pipelines 

In a continuous-delivery environment, many pipelines can contain identical components, except for parameters. Managed pipeline templates provide a perfect way to reduce code duplication and centralize pipeline management, affecting hundreds or thousands of pipelines downstream. This method reduces mistakes, saves time and allows the same template-based configuration to provision multiple identical environments, such as staging, QA and production.

Automated checks and canary analysis on each infrastructure change 

You’re probably familiar with using deployment pipelines to manage application and configuration changes, but now you can use them for infrastructure changes, too — changed instance types, firewall rules, load balancers and so on. Having an official deployment pipeline for each application means you can now run automated testing and canary analysis to even these changes, making infrastructure changes much safer.

Defining a clear paved road for deployments + override capabilities by each application 

We've found that most deployment pipelines follow a very simple pattern:
However, most teams customize these pipelines based on their requirements. Here are just a few examples:

• Resize previous group to zero instead of destroying. 
• Lots of parameters for the deploy stage, wait stage. 
• Two canary analysis stages (first with 1% of traffic, the next with 50% of traffic). 
• Various parameters passed on to the canary analysis. 

Pipeline templates allow these teams to start off with a paved road, then mix and match stages, specify parameters, and replace or inject stages as needed, while still using the maintained paved road template. If a common stage in the base template is updated, the pipelines inherit the updates automatically. This reduces maintenance considerably for hundreds of pipelines.

Conditional stages use variables to control which stages are enabled. By utilizing conditional stages, you can use a single template for more than one use case. Here’s a video demo showing how it looks in practice.

Injecting stages allows any pipeline child template or configuration to add stages to the pipeline stage graph. For example, if a team uses a basic Bake -> Deploy -> Disable -> Wait -> Destroy pipeline template, they can easily inject a Manual Judgement stage to require a human decision before the previous group is disabled and destroyed. Here’s how this looks in practice.

Automatic deployment pipeline creation for new services 

When a new service comes online — whether in the development, staging or production environments — pipeline templates can be used to create automatic deployment pipelines as a starting point. This reduces the effort of each team as they get a tested and fully working deployment pipeline out of the box (which can later on be customized if needed). Traffic guards and canary analysis give us the confidence to do this completely automatically.

Auto-generate pipelines to perform a rolling OS upgrade across all applications 

Applying OS security updates safely across an entire large scale production system can be quite a challenge. Doing so while keeping the fleet immutable is even harder. Spinnaker pipeline templates combined with canary analysis can provide organizations with a framework to automate this task.

We use a weekly OS upgrade pipeline, which spawns the official deployment pipelines of all production applications, in a rolling manner, from least critical to most critical, spreading the upgrades to small batches which are then cascaded across the entire work week. Each iteration of the pipeline upgrades more applications than it did the day before. We use the official deployment pipelines for each application, sending a runtime parameter which says “Don’t change the binary or configuration — just rebake the base operating system with latest security updates,” and we get a new immutable image unchanged from the previous image except for those security updates. All this while still going through the usual canary analysis safeguards, load balancer health checks and traffic guards.

Pipeline templates can take this pattern one step further. Any new application can be automatically added to this main OS upgrade pipeline, ensuring the entire production fleet is always up to date for OS security updates.

Conclusion 

Spinnaker pipeline templates solve a major issue for organizations running a lot of deployment pipelines. Plus, being able to control infrastructure as code for multiple providers, having it version-controlled and living alongside the application and configuration, removes a major constraint and could be a big step forward in taming operational chaos.

Getting-started references 

To get started with pipeline templates:

• Set up Spinnaker. 
• Read the spec, review the pipeline templates getting started guide. 
• Check out the converter from existing pipelines into templates, a good place to start. 
• roer, the spinnaker thin CLI used to validate and publish templates and configurations to Spinnaker. (Note: Better tooling is on the way.) Remember: roer pipeline-template plan is your friend. Also, orca.log is a great place to debug after publishing a template. 
• Watch the full demo. 
• Try out some example templates. 
• Join the Spinnaker slack community and ask questions on the #declarative-pipelines channel.