Editor’s note: This is the second in a series of blog posts on container security at Google.
When deploying containers, your container images should be free of known vulnerabilities, and have a bare minimum of functionality. This reduces the attack surface, preventing bad actors from taking advantage of unnecessary openings in your infrastructure.
Unlike other deployment mechanisms, with containers, there are actually two operating systems that you need to secure—the container’s node/host OS, i.e., the operating system on which you run the container; and the container image that runs inside the container itself. On Google Kubernetes Engine, our managed container service, as well as for other hosted services on Google Cloud Platform (GCP), we manage the node OS for you. And when it comes to the container image, we give you several options to choose from.
Out of the box, Kubernetes Engine provides the following options for your node OS and container images:
- For the node OS, you can choose between Container-optimized OS (COS) or Ubuntu
- For the container image, Google Container Registry has readily available images for Debian and Ubuntu, and of course, you can also bring your own image!
It’s great to have choices—but choice can also be overwhelming. Let’s take a deeper look at the security properties of these options, and what’s included in Kubernetes Engine.
Node OS: Container-optimized OS (COS)
Container-optimized OS (COS) is a relatively new OS that we developed to enhance the security and performance of services running in Google Cloud, especially containers. In fact, COS underpins Kubernetes Engine, Cloud SQL, Cloud Machine Learning Engine and several other Google services.Based on Chromium OS, COS implements several security design principles to provide a manageable platform for running production services. Some of these design aspects include:
- Minimal OS footprint. COS is optimized to run containers on GCP. As such, we only enable features and include packages that are absolutely necessary to support running containers. Since containers package their own dependencies, this allows us to greatly reduce the OS attack surface and also improves performance.
- Read-only root filesystem. The COS root filesystem is always mounted as read-only. Additionally, its checksum is verified by the kernel on each boot. This means that the kernel refuses to boot if the root filesystem has been tampered with. Additionally, several other mounts are non-executable by default.
- Stateless configuration. While having a read-only root filesystem is good for security, it makes the system unusable for all practical purposes (e.g., we need to be able to create and add users in order to log in to the system). To address this, we customized the root filesystem such that /etc/ is stateless. This allows you to write configuration settings at run time, but those settings do not persist across reboots. Thus, every time a COS node reboots, it starts from a clean slate. Certain areas, such as users’ home directories, logs, and docker images, persist across reboots, as they're not part of the root filesystem.
- Security-hardened kernel. COS enables several security-hardening kernel features, including some from the ChromiumOS Linux Security Module (LSM). For example, by using a combination of LoadPin (one such LSM that comes from ChromiumOS) and the read-only rootfs and rootfs-verification, you can prevent attackers from compromising the kernel by loading custom kernel modules. Additionally, Linux features such as IMA, AUDIT, APPARMOR, etc. make it difficult to hide attempts at circumventing security policies.
- Sane security-centric defaults. COS provides another level of hardening simply by providing sane default values for several features. This includes things such as sysctl settings that disable ptrace and unprivileged bpf, a locked down firewall, and so on. These sane defaults, when automatically applied to a fleet of instances, go a long way toward securing the entire cluster/project/organization.
- Auto-updates. COS’s automatic updates feature allows timely delivery of security patches to running VMs. When COS is managed by Kubernetes Engine, Node-Auto-Upgrade strikes a balance between security and stability.
In addition to various hardening features in the OS itself, the COS team also employs best practices when developing, building and deploying these OS images to Google Cloud. Some of these include:
- Built from source at Google. Each package in COS, including the Linux kernel itself, is built from source from ChromiumOS code repositories. This means that we know exactly what is going into the OS, who checked it in, in which version it was introduced, etc. This also lets us quickly patch and update any package in case a vulnerability is discovered, at any level.
- Continuous vulnerability scanning and response. A CVE-scanning system alerts us whenever a vulnerability is discovered in any component of the OS. The COS team then responds with priority to make patched releases available for our users. The COS team also works with Google’s incident response team to make wider security patches available quickly in COS, e.g., patched COS images were available on Google Cloud before the recent Spectre and Meltdown vulnerabilities were publicly announced.
- Testing and qualification process. Before a new COS image is published to Google Cloud, it undergoes extensive testing at multiple levels—including kernel fuzz testing by syzkaller, cluster-level Kubernetes tests, and several performance benchmarks. This ensures the stability and quality of our releases.
We are also actively working on several improvements in the area of node-OS security. You can learn more in the COS security documentation.
Kubernetes Engine uses COS as the OS for all master nodes. By default, COS is also used for your workload’s node OS. Unless you have specific requirements, we recommend you use COS for its security properties.
Container image: Debian and Ubuntu
Similarly to our node OS, we maintain our own container images for running hosted services. Google Cloud uses Debian and Ubuntu as a base image, for services like Google App Engine or Google Cloud Functions. Likewise, Debian and Ubuntu are both popular choices for container images.
From a security perspective, it doesn’t matter which container image you use, the important thing is to scan it regularly for known vulnerabilities. We maintain our Debian and Ubuntu base images with regular patching and testing and can rebuild them from scratch reproducibly. If you’re building your own containers, you’re welcome to use our base images too!
- Debian is available on Google Container Registry
- Ubuntu is available on Google Container Registry
