Tag Archives: packaging

Kpt: Packaging up your Kubernetes configuration with git and YAML since 2014

Kubernetes configuration manifests have become an industry standard for deploying both custom and off-the-shelf applications (as well as for infrastructure). Manifests are combined into bundles to create higher-level deployable systems as well as reusable blueprints (such as a product offering, off the shelf software, or customizable starting point for a new application).

However, most teams lack the expertise or desire to create bespoke bundles of configuration from scratch and instead: 1) either fork them from another bundle, or 2) use some packaging solution which generates manifests from code.

Teams quickly discover they need to customize, validate, audit and re-publish their forked/ generated bundles for their environment. Most packaging solutions to date are tightly coupled to some format written as code (e.g. templates, DSLs, etc). This introduces a number of challenges when trying to extend, build on top of, or integrate them with other systems. For example, how does one update a forked template from upstream, or how does one apply custom validation?

Packaging is the foundation of building reusable components, but it also incurs a productivity tax on the users of those components.

Today we’d like to introduce kpt, an OSS tool for Kubernetes packaging, which uses a standard format to bundle, publish, customize, update, and apply configuration manifests.

Kpt is built around an “as data” architecture bundling Kubernetes resource configuration, a format for both humans and machines. The ability for tools to read and write the package contents using standardized data structures enables powerful new capabilities:
  • Any existing directory in a Git repo with configuration files can be used as a kpt package.
  • Packages can be arbitrarily customized and later pull in updates from upstream by merging them.
  • Tools and automation can perform high-level operations by transforming and validating package data on behalf of users or systems.
  • Organizations can develop their own tools and automation which operate against the package data.
  • Existing tools and automation that work with resource configuration “just work” with kpt.
  • Existing solutions that generate configuration (e.g. from templates or DSLs) can emit kpt packages which enable the above capabilities for them.

Example workflow with kpt

Now that we’ve established the benefits of using kpt for managing your packages of Kubernetes config, lets walk through how an enterprise might leverage kpt to package, share and use their best practices for Kubernetes across the organization.

First, a team within the organization may build and contribute to a repository of best practices (pictured in blue) for managing a certain type of application, for example a microservice (called “app”). As the best practices are developed within an organization, downstream teams will want to consume and modify configuration blueprints based on them. These blueprints provide a blessed starting point which adheres to organization policies and conventions.

The downstream team will get their own copy of a package by downloading it to their local filesystem (pictured in red) using kpt pkg get. This clones the git subdirectory, recording upstream metadata so that it can be updated later.

They may decide to update the number of replicas to fit their scaling requirements or may need to alter part of the image field to be the image name for their app. They can directly modify the configuration using a text editor (as would be done before). Alternatively, the package may define setters, allowing fields to be set programmatically using kpt cfg set. Setters streamline workflows by providing user and automation friendly commands to perform common operations.

Once the modifications have been made to the local filesystem, the team will commit and push their package to an app repository owned by them. From there, a CI/CD pipeline will kick off and the deployment process will begin. As a final customization before the package is deployed to the cluster, the CI/CD pipeline will inject the digest of the image it just built into the image field (using kpt cfg set). When the image digest has been set, the CI/CD pipeline can send the manifests to the cluster using kpt live apply. Kpt live operates like kubectl apply, providing additional functionality to prune resources deleted from the configuration and block on rollout completion (reporting status of the rollout back to the user).

Now that we’ve walked through how you might use kpt in your organization, we’d love it if you’d try it out, read the docs, or contribute.

One more thing

There’s still a lot to the story we didn’t cover here. Expect to hear more from us about:
  • Using kpt with GitOps
  • Building custom logic with functions
  • Writing effective blueprints with kpt and kustomize
By Phillip Wittrock, Software Engineer and Vic Iglesias, Cloud Solutions Architect

Understanding APK packaging in Android Studio 2.2

Posted by Wojtek Kaliciński, Android Developer Advocate

Android Studio 2.2 launched recently with many new and improved features. Some of the changes are easy to miss because they happened under the hood in the Android Gradle plugin, such as the newly rewritten integrated APK packaging and signing step.

APK Signature Scheme v2

With the introduction of the new APK Signature Scheme v2 in Android 7.0 Nougat, we decided to rewrite how assembling APKs works in the Android Gradle plugin. You can read all about the low-level technical details of v2 signatures in the documentation, but here's a quick tl;dr summary of the info you need as an Android app developer:

  • The cryptographic signature of the APK that is used to verify its integrity is now located immediately before the ZIP Central Directory.
  • The signature is computed and verified over the binary contents of the whole APK file, as opposed to decompressed file contents of each file in the archive in v1.
  • An APK can be signed by both v1 and v2 signatures at the same time, so it remains backwards compatible with previous Android releases.

Why introduce this change to how Android verifies APKs? Firstly, for enhanced security and extensibility of this new signing format, and secondly for performance - the new signatures take significantly less time to verify on the device (no need for costly decompression), resulting in faster app installation times.

The consequence of this new signing scheme, however, is that there are new constraints on the APK creation process. Since only uncompressed file contents were verified in v1, that allowed for quite a lot of modifications to be made after APK signing - files could be moved around or even recompressed. In fact, the zipalign tool which was part of the build process did exactly that - it was used to align ZIP entries on correct byte boundaries for improved runtime performance.

Because v2 signatures verify all bytes in the archive and not individual ZIP entries, running zipalign is no longer possible after signing. That's why compression, aligning and signing now happens in a single, integrated step of the build process.

If you have any custom tasks in your build process that involve tampering with or post-processing the APK file in any way, please make sure you disable them or you risk invalidating the v2 signature and thus making your APKs incompatible with Android 7.0 and above.

Should you choose to do signing and aligning manually (such as from the command line), we offer a new tool in the Android SDK, called apksigner, that provides both v1 and v2 APK signing and verification. Note that you need to run zipalign before running apksigner if you are using v2 signatures. Also remember the jarsigner tool from the JDK is not compatible with Android v2 signatures, so you can't use it to re-sign your APKs if you want to retain the v2 signature.

In case you want to disable adding v1 or v2 signatures when building with the Android Gradle plugin, you can add these lines to your signingConfig section in build.gradle:

v1SigningEnabled false
v2SigningEnabled false

Note: both signing schemes are enabled by default in Android Gradle plugin 2.2.

Release builds for smaller APKs

We took this opportunity when rewriting the packager to make some optimizations to the size of release APKs, resulting in faster downloads, smaller delta updates on the Play Store, and less wasted space on the device. Here are some of the changes we made:

  • Files in the archive are now sorted to minimize differences between APK builds.
  • All file timestamps and metadata are zeroed out.
  • Level 6 and level 9 compression is checked for all files in parallel and the optimal one is used, i.e. if L9 provides little benefit in terms of size, then L6 may be chosen for better performance
  • Native libraries are stored uncompressed and page aligned in the APK. This brings support for the android:extractNativeLibs="false" option from Android 6.0 Marshmallow and lets applications use less space on the device as well as generate smaller updates on the Play Store
  • Zopfli compression is not used to better support Play Store update algorithms. It is not recommended to recompress your APKs with Zopfli. Pre-optimizing individual resources such as PNG files in your projects is still fine and recommended.

These changes help make your releases as small as possible so that users can download and update your app even on a slower connection or on less capable devices. But what about debug builds?

Debug builds for installation speed

When developing apps you want to keep the iteration cycle fast - change code, build, and deploy on a connected device or emulator. Since Android Studio 2.0 we've been working to make all the steps as fast as possible. With Instant Run we're now able to update only the changed code and resources during runtime, while the new Emulator brings multi-processor support and faster ADB speeds for quicker APK transfer and installation. Build improvements can cut that time even further and in Android Studio 2.2 we're introducing incremental packaging and parallel compression for debug builds. Together with other features like selectively packaging resources for the target device density and ABI this will make your development even faster.

A word of caution: the APK files created for Instant Run or by invoking a debug build are not meant for distribution on the Play Store! They contain additional instrumentation code for Instant Run and are missing resources for device configurations other than the one that was connected when you started the build. Make sure you only distribute release versions of the APK which you can create using the Android Studio Generate Signed APK command or the assembleRelease Gradle task.