Tag Archives: codelabs

Migrating from App Engine Users to Cloud Identity Platform (Module 21)

Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

The Serverless Migration Station series is aimed at helping developers modernize their apps running one of Google Cloud's serverless platforms. The preceding (Migration Module 20) video demonstrates how to add use of App Engine's Users service to a Python 2 App Engine sample app. Today's Module 21 video picks up from where that leaves off, migrating that usage to Cloud Identity Platform.
How to migrate the App Engine Users to Cloud Identity Platform
Moving away from proprietary App Engine bundled services like Users makes apps more portable, giving them enough flexibility to:

    Understanding the overall migration

    Overall, Module 21 features major changes to the Module 20 sample app, implementing a move from App Engine bundled services (NDB & Users) to standalone Cloud services (Cloud Datastore & Identity Platform). Identity Platform doesn't know anything about App Engine admins, so that must be built, requiring the use of the Cloud Resource Manager API. Apps dependent on Python 2 have additional required updates. Let's discuss in a bit more detail.

    Migration "parts"

    The following changes to the sample app are required:

    • Migrate from App Engine Users (server-side) to Cloud Identity Platform (client-side)
    • Migrate from App Engine NDB, the other bundled service used in Module 20, to Cloud NDB (requires use of the Cloud Datastore API)
    • Use the Cloud Resource Manager* (via its API) to fetch the Cloud project's IAM allow policy to collate the set of App Engine admin users for the app.
    • Use the Firebase Admin SDK to validate whether the user is an App Engine admin
    • Migrate from Python 2 to 3 (and possibly back to Python 2 [more on this below])
     
    *At the time of this writing, the Resource Manager documentation only features setup instructions for accessing the API from the lower-level Google APIs client library rather than the Resource Manager client library. To learn how to set up the latter, go to the Resource Manager client library documentation directly. The lower-level client library should only be used in circumstances when a Cloud client library doesn't exist or doesn't have the features your app needs. One such use case is Python 2, and we'll be covering that shortly.
     

      Move from App Engine bundled services to standalone Cloud services

      The NDB to Cloud NDB migration is identical to the Module 2 migration content, so it's not covered in-depth here in Module 21. The primary focus is on switching to Identity Platform to continue supporting user logins as well as implementing use of the Resource Manager and Firebase Admin SDK to build a proxy for recognizing App Engine admin users as provided by the Users service. Below is pseudocode implementing the key changes to the main application where new or updated lines of code are bolded:

      Table showing changes in code 'Before'(Module 20) and 'After'(Module 21)
      Migrating from App Engine Users to Cloud Identity Platform(click to enlarge)

      The key differences to note:

      1. The server-side Users service code vanishes from the main application, moving into the (client-side) web template (not shown here).
      2. Practically all of the new code in the Module 21 app above is for recognizing App Engine admin users. There are no changes to app operations or data models other than Cloud NDB requiring use of Python context managers to wrap all Datastore code (using Python with blocks).

      Complete versions of the app before and after the updates can be found in the Module 20 (Python 2) and Module 21 (Python 3) repo folders, respectively. In addition to the video, be sure to check out the Identity Platform documentation as well as the Module 21 codelab which leads you step-by-step through the migrations discussed.

      Aside from the necessary coding changes as well as moving from server-side to client-side, note that the Users service usage is covered by App Engine's pricing model while Identity Platform is an independent Cloud service billed by MAUs (monthly active users), so costs should be taken into account if migrating. More information can be found in the Identity Platform pricing documentation.

      Python 2 considerations

      With the sunset of Python 2, Java 8, PHP 5, and Go 1.11, by their respective communities, Google Cloud has assured users by expressing continued long-term support of these legacy App Engine runtimes, including maintaining the Python 2 runtime. So while there is no current requirement for users to migrate, developers themselves are expressing interest in updating their applications to the latest language releases.
      The primary Module 21 migration automatically includes a port from Python 2 to 3 as that's where most developers are headed. For those with dependencies requiring remaining on Python 2, some additional effort is required:


        The codelab covers this backport in-depth, so check out the specific section for Python 2 users if you're in this situation. If you don't want to think about it, just head to the repo for a working Python 2 version of the Module 21 app.

        Wrap-up

        Module 21 features migrations of App Engine bundled services to appropriate standalone Cloud services. While we recommend users modernize their App Engine apps by moving to the latest offerings from Google Cloud, these migrations are not required. In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you don't have to migrate to standalone services before porting your app to Python 3. You can continue using App Engine NDB and Users in Python 3 so long as you retrofit your code to access bundled services from next-generation runtimes. Then should you opt to migrate, you can do so on your own timeline.

        If you're using other App Engine legacy services be sure to check out the other Migration Modules in this series. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, the Cloud team is working on covering other language runtimes, so stay tuned. For additional video content, check out our broader Serverless Expeditions series.

        How to use the App Engine Users service (Module 20)

        Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud


        Introduction and background

        The Serverless Migration Station video series and corresponding codelabs aim to help App Engine developers modernize their apps, whether it's upgrading language runtimes like from Python 2 to 3 and Java 8 to 17, or to move laterally to sister serverless platforms like Cloud Functions or Cloud Run. For developers who want more control, like being able to SSH into instances, Compute Engine VMs or GKE, our managed Kubernetes service, are also viable options.

        In order to consider moving App Engine apps to other compute services, developers must move their apps away from its original APIs (now referred to as legacy bundled services), either to Cloud standalone replacement or alternative 3rd-party services. Once no longer dependent on these proprietary services, apps become much more portable. Apps can stay on App Engine while upgrading to its 2nd-generation platform, or move to other compute platforms as listed above.

        Today's Migration Module 20 content focuses on helping developers refamiliarize themselves with App Engine's Users service, a user authentication system serving as a lightweight wrapper around Google Sign-In (now called Google Identity Services). The video and its corresponding codelab (self-paced, hands-on tutorial) demonstrate how to add use of the Users service to the sample baseline app from Module 1. After adding the Users service in Module 20, Module 21 follows, showing developers how to migrate that usage to Cloud Identity Platform.

        How to use the App Engine Users service

        Adding use of Users service


        The sample app's basic functionality consists of registering each page visit in Datastore and displaying the most recent visits. The Users service helps apps support user logins, App Engine administrative ("admin'") users. It also provides convenient functions for generating login/logout links and retrieving basic user information for logged-in users. Below is a screenshot of the modified app which now supports user logins via the user interface (UI):
        Sample app now supports user logins and App Engine admin users (click to enlarge) 
        Below is the pseudocode reflecting the changes made to support user logins for the sample app, including integrating the Users service and updating what shows up in the UI:
        • If the user is logged in, show their "nickname" (display name or email address) and display a Logout button. If the logged-in user is an App Engine app admin, also display an "admin" badge (between nickname and Logout button).
        • If the user is not logged in, display the username generically as "user", remove any admin badge, and display a Login button.
        Because the Users service is primarily a user-facing endeavor, the most significant changes take place in the UI, whereas the data model and core functionality of registering visits remain unchanged. The new support for user management primarily results in additional context to be rendered in the web template. New or altered code is bolded to highlight the updates.
        Table showing code 'Before'(Module 1) on left, and 'After' (Module 20) on the right
         Adding App Engine Users service usage to sample app (click to enlarge)

        Wrap-up


        Today's "migration" consists of adding usage of the App Engine Users service to support user management and recognize App Engine admin users, starting with the Module 1 baseline app and finishing with the Module 20 app. To get hands-on experience doing it yourself, try the codelab and follow along with the video. Then you'll be ready to upgrade to Identity Platform should you choose to do so.

        In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you are no longer required to migrate from the Users service to Identity Platform when porting your app to Python 3. You can continue using the Users service in your Python 3 app so long as you retrofit the code to access bundled services from next-generation runtimes.

        If you do want to move to Identity Platform, see the Module 21 content, including its codelab. All Serverless Migration Station content (codelabs, videos, and source code [when available]) are available at its open source repo. While we're initially focusing on Python users, the Cloud team is covering other runtimes soon, so stay tuned. Also check out other videos in the broader Serverless Expeditions series.

        Migrating from App Engine pull tasks to Cloud Pub/Sub (Module 19)

        Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

        Introduction and background

        The Serverless Migration Station series is aimed at helping developers modernize their apps running one of Google Cloud's serverless platforms. The preceding (Migration Module 18) video demonstrates how to add use of App Engine's Task Queue pull tasks service to a Python 2 App Engine sample app. Today's Module 19 video picks up from where that leaves off, migrating that pull task usage to Cloud Pub/Sub.

        Moving away from proprietary App Engine services like Task Queue makes apps more portable, giving them enough flexibility to:

         

          Understanding the migrations

          Module 19 consists of implementing three different migrations on the Module 18 sample app:

          • Migrate from App Engine NDB to Cloud NDB
          • Migrate from App Engine Task Queue pull tasks to Cloud Pub/Sub
          • Migrate from Python 2 to Python (2 and) 3

          The NDB to Cloud NDB migration is identical to the Module 2 migration content, so it's not covered in-depth in Module 19. The original app was designed to be Python 2 and 3 compatible, so there's no work there either. Module 19 boils down to three key updates:

          • Setup: Enable APIs and create Pub/Sub Topic & Subscription
          • How work is created: Publish Pub/Sub messages instead of adding pull tasks
          • How work is processed: Pull messages instead of leasing tasks

          Aside from these physical changes, a key hurdle to overcome is understanding the differences in terminology between pull tasks and Pub/Sub. The following chart attempts to demystify this so developers can more easily grasp how they differ:
          Table of terminology with related GAE Pull Tasks and Cloud Pub/Sub
          Terminology differences between App Engine pull tasks and Cloud Pub/Sub

          Reflecting the chart, these differences can be summarized like this:
          1. With Pull Queues, work is created in pull queues while work is sent to Pub/Sub topics
          2. Task Queue pull tasks are called messages in Pub/Sub
          3. With Task Queues, workers access pull tasks; with Pub/Sub, subscribers receive messages
          4. Leasing a pull task is the same as pulling a message from a Pub/Sub topic via a subscription
          5. Deleting a task from a pull queue when you're done is analogous to successfully acknowledging a Pub/Sub message
          The video walks developers through the terminology as well as the code changes described above. Below is pseudocode implementing the key changes to the main application (new or updated lines of code bolded):
          Table showing changes in code Before (Module 18) on the left, and After (Module 19) on the right
          Migration from App Engine Task Queue pull tasks to Cloud Pub/Sub

          Observe how most of the code, especially app operations and data models are left relatively unchanged. The only visible changes are switching from App Engine NDB and Task Queue to Cloud NDB and Pub/Sub. Complete versions of the app before and after making the changes can be found in the Module 18 and Module 19 repo folders, respectively. In addition to the video, be sure to check out the Module 19 codelab which leads you step-by-step through the migrations discussed.

          Wrap-up

          Module 19 features a migration of App Engine pull tasks to Cloud Pub/Sub, but developers should note that Pub/Sub itself is not based on pull tasks. It is a fully-featured asynchronous, scalable messaging service that has many more features than the pull functionality provided by Task Queue. For example, Pub/Sub has other features like streaming to BigQuery and push functionality. Pub/Sub push operates differently than Task Queue push tasks, hence why we recommend push tasks be migrated to Cloud Tasks instead (see Module 8). For more information on all of its features, see the Pub/Sub documentation. Because Cloud Tasks doesn't support pull functionality, we turn to Pub/Sub instead for pull task users.

          While we recommend users move to the latest offerings from Google Cloud, neither of those migrations are required, and should you opt to do so, can do them on your own timeline. In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you don't have to migrate to standalone Cloud services before porting your app to Python 3. You can continue using Task Queue in Python 3 so long as you retrofit your code to access bundled services from next-generation runtimes.

          If you're using other App Engine legacy services be sure to check out the other Migration Modules in this series. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, the Cloud team is working on covering other language runtimes, so stay tuned. For additional video content, check out our broader Serverless Expeditions series.

          How to use App Engine pull tasks (Module 18)

          Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

          Introduction and background

          The Serverless Migration Station mini-series helps App Engine developers modernize their apps to the latest language runtimes, such as from Python 2 to 3 or Java 8 to 17, or to sister serverless platforms Cloud Functions and Cloud Run. Another goal of this series is to demonstrate how to move away from App Engine's original APIs (now referred to as legacy bundled services) to Cloud standalone replacement services. Once no longer dependent on these proprietary services, apps become much more portable, making them flexible enough to:

          App Engine's Task Queue service provides infrastructure for executing tasks outside of the standard request-response workflow. Tasks may consist of workloads exceeding request timeouts or periodic tangential work. The Task Queue service provides two different queue types, push and pull, for developers to perform auxiliary work.

          Push queues are covered in Migration Modules 7-9, demonstrating how to add use of push tasks to an existing baseline app followed by steps to migrate that functionality to Cloud Tasks, the standalone successor to the Task Queues push service. We turn to pull queues in today's video where Module 18 demonstrates how to add use of pull tasks to the same baseline sample app. Module 19 follows, showing how to migrate that usage to Cloud Pub/Sub.

          Adding use of pull queues

          In addition to registering page visits, the sample app needs to be modified to track visitors. Visits are comprised of a timestamp and visitor information such as the IP address and user agent. We'll modify the app to use the IP address and track how many visits come from each address seen. The home page is modified to show the top visitors in addition to the most recent visits:

          Screen grab of the sample app's updated home page tracking visits and visitors
          The sample app's updated home page tracking visits and visitors

          When visits are registered, pull tasks are created to track the visitors. The pull tasks sit patiently in the queue until they are processed in aggregate periodically. Until that happens, the top visitors table stays static. These tasks can be processed in a number of ways: periodically by a cron or Cloud Scheduler job, a separate App Engine backend service, explicitly by a user (via browser or command-line HTTP request), event-triggered Cloud Function, etc. In the tutorial, we issue a curl request to the app's endpoint to process the enqueued tasks. When all tasks have completed, the table then reflects any changes to the current top visitors and their visit counts:

          Screen grab of processed pull tasks updated in the top visitors table
          Processed pull tasks update the top visitors table

          Below is some pseudocode representing the core part of the app that was altered to add Task Queue pull task usage, namely a new data model class, VisitorCount, to track visitor counts, enqueuing a (pull) task to update visitor counts when registering individual visits in store_visit(), and most importantly, a new function fetch_counts(), accessible via /log, to process enqueued tasks and update overall visitor counts. The bolded lines represent the new or altered code.

          Adding App Engine Task Queue pull task usage to sample app showing 'Before'[Module 1] on the left and 'After' [Module 18] with altered code on the right
          Adding App Engine Task Queue pull task usage to sample app

          Wrap-up

          This "migration" is comprised of adding Task Queue pull task usage to support tracking visitor counts to the Module 1 baseline app and arrives at the finish line with the Module 18 app. To get hands-on experience doing it yourself, do the codelab by hand and follow along with the video. Then you'll be ready to upgrade to Cloud Pub/Sub should you choose to do so.

          In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you are no longer required to migrate pull tasks to Pub/Sub when porting your app to Python 3. You can continue using Task Queue in your Python 3 app so long as you retrofit the code to access bundled services from next-generation runtimes.

          If you do want to move to Pub/Sub, see Module 19, including its codelab. All Serverless Migration Station content (codelabs, videos, and source code) are available at its open source repo. While we're initially focusing on Python users, the Cloud team is covering other runtimes soon, so stay tuned. Also check out other videos in the broader Serverless Expeditions series.

          Per-App Language Preferences – Part 1

          Posted by Neelansh Sahai Android Developer Relations Engineer (on Twitter and LinkedIn)What if you have a set of users who are quite fluent in English, Hindi, and Spanish, and they have a news app on their phones and prefer to read the news in Hindi? For their texting app, they prefer Spanish as they have some friends and family who they text with in Spanish. But for ease of access, they still prefer their device to be in English. Now there are many such use-cases where the users might want their app languages to be different from their system language. Interesting!

          Starting with Android 13, we have included one of the most-requested features from users, Per-App Language Preferences. It lets users switch app languages from System settings, providing users with better control over their language choices for different apps, irrespective of the system language.
          A cellphone screen displaying App language preferences in system settings for the YouTube app

          Build for your multilingual users

          This blog focuses on how to integrate the Per-App Language Preferences API in your app to provide your users the flexibility to choose different languages for different apps.

          1.    Users can change the language settings from system settings by selecting: 

          Settings → System → Languages & Input → App Languages → [Select the desired App] → [Select the desired Language]

          NOTE: Only those apps that have opted in for the feature by specifying the locale_config.xml file (more on this below), will appear in system settings.

          A cellphone screen demonstrating finding the language settings from system settings by selecting Settings → System → Languages & Input → App Languages → [Select the desired App] → [Select the desired Language]
          2.    If your app already has an in-app language picker, you can integrate the Per-App Language Preferences API to leverage the full platform support. For pre-Android 13 users, the system settings won’t be visible, but developers can still provide an in-app language picker.

          A cellphone screen demonstrating integrating the Per-App Language prefences API for an app which already has an in-app language picker

          How to integrate this feature in your app?

          There are 5 steps that need to be followed while working on the Per-App Language Preferences feature, listed here →

           


          1.    Create locale_config.xml file


          Create a new file in values/xml/ directory and name it as locale_config.xml. This file should contain a list of all the locales that are supported by the app. The list element should be a string containing a locale tag.

          NOTE: The locale tags must follow the BCP47 syntax, which is usually {language subtag}–{script subtag}–{country subtag}. Anything other than that will be filtered out by the system and won't be visible in the system settings.


          locale_config.xml

          <?xml version="1.0" encoding="utf-8"?>
          <locale-config xmlns:android="http://schemas.android.com/apk/res/android">
              ...

              <!-- English -->
              <locale android:name="en"/>



              <!-- Japanese (Japan) -->          
              <locale android:name="ja-JP"/>



              <!-- Chinese (Macao) in Simplified Script -->
              <locale android:name="zh-Hans-MO"/>



              <!-- Chinese (Taiwan) in Traditional Script -->
              <locale android:name="zh-Hant-TW"/>  
              ...
          </locale-config>





          2.    Add the locale_config in the AndroidManifest.xml

          Specify this locale_config.xml file in the app’s AndroidManifest.xml

          AndroidManifest.xml

          <manifest>
              ...
              <application
                  ...
                  android:localeConfig="@xml/locale_config">
              </application>
          </manifest>


          After steps 1 & 2, your users will be able to discover and set their language preference for your app from system settings on devices running Android 13 or higher. In case your users are on devices running on versions lower than Android 13, you can provide an in-app language picker. Optionally, you can also include the same language picker in your app for devices running Android 13 or higher. When your app includes an in-app language picker, it's important for the user's preferences to be in sync between the system and the app. This is where the AndroidX APIs come into the picture. Read on to learn how to create an in-app language picker.




          3. Add the libraries

          Use the latest version of AppCompat Library

          build.gradle (app)

          def latestAppCompatVersion =  “1.6.0-rc01”

          dependencies {
              ...
              implementation "androidx.appcompat:appcompat:$latestAppCompatVersion"
              implementation "androidx.appcompat:appcompat-resources:$latestAppCompatVersion"
              ...
          }




          4. Use AndroidX APIs

          Use the APIs in your code to set and get the app locales.

          MainActivity.kt

          val appLocale: LocaleListCompat = LocaleListCompat.forLanguageTags("xx-YY")

          // Call this on the main thread as it may require Activity.restart()
          AppCompatDelegate.setApplicationLocales(appLocale)


          // Call this to get the selected locale and display it in your App
          val selectedLocale = AppCompatDelegate.getApplicationLocales()[0]

          NOTE: These APIs are also backward compatible, so even if the app is being used on Android 12 or lower, the APIs would still behave the same, and no additional checks for OS versions are required in your code.


           
          5. Delegate storage to AndroidX

          Let AndroidX handle the locale storage so that the user's preference persists.

          AndroidManifest.xml

          <application
              ...
              <service
                  android:name="androidx.appcompat.app.AppLocalesMetadataHolderService"
                  android:enabled="false"
                  android:exported="false">
                  <meta-data
                      android:name="autoStoreLocales"
                      android:value="true" />
              </service>
              ...
          </application>


          Steps 3, 4, & 5 above demonstrate the minimum components needed to create an in-app language picker.

          And with this, your app can now support locale switching.


          Additional things to take care of while migrating to the API

          Earlier, developers had to handle the user's preferences on their own, either by using SharedPreferences, storing the data on a server, or other app logic. With the new APIs, there is no need to handle this separately. So when you are using these APIs, AndroidX is already taking care of the storage part, but what happens when the app is opened for the first time after a user updates their device to Android 13 or higher?

          In this case, the system won’t be aware of the user’s preferences for the app language and thus it will map the app to the default system language. To avoid this, developers need to add some one-time migration logic so that their users don’t have to set the language again when they update the app.

          // Specify the constants to be used in the below code snippets

          companion object {

              // Constants for SharedPreference File
              const val PREFERENCE_NAME = "shared_preference"
              const val PREFERENCE_MODE = Context.MODE_PRIVATE

              // Constants for SharedPreference Keys
              const val FIRST_TIME_MIGRATION = "first_time_migration"
              const val SELECTED_LANGUAGE = "selected_language"

              // Constants for SharedPreference Values
              const val STATUS_DONE = "status_done"
          }




          // Utility method to put a string in a SharedPreference
          private fun putString(key: String, value: String) {
              val editor = getSharedPreferences(PREFERENCE_NAME, PREFERENCE_MODE).edit()
              editor.putString(key, value)
              editor.apply()
          }

          // Utility method to get a string from a SharedPreference
          private fun getString(key: String): String? {
              val preference = getSharedPreferences(PREFERENCE_NAME, PREFERENCE_MODE)
              return preference.getString(key, null)
          }


          // Check if the migration has already been done or not
          if (getString(FIRST_TIME_MIGRATION) != STATUS_DONE) {

             // Fetch the selected language from wherever it was stored. In this case it’s SharedPref

             // In this case let’s assume that it was stored in a key named SELECTED_LANGUAGE
             getString(SELECTED_LANGUAGE)?.let { it →

                // Set this locale using the AndroidX library that will handle the storage itself
                val localeList = LocaleListCompat.forLanguageTags(it)
                AppCompatDelegate.setApplicationLocales(localeList)

                // Set the migration flag to ensure that this is executed only once
                putString(FIRST_TIME_MIGRATION, STATUS_DONE)
             }
          }

           

          What flexibility does the feature provide to the users and developers?

          Here are a few things that might prove to be beneficial for you users.

          1. All devices running Android 13 or higher will have a common place for users to discover and change the language of their apps.
          2. Although the system settings are limited to the devices running Android 13 or higher, the AndroidX APIs are backwards compatible. Thus, there is no requirement to add OS Version checks in your code while building for your multilingual users.
          3. Developers do not need to handle configuration changes separately or worry about storing the user's selected language every time. The API handles configuration changes and stores the language preferences for you.
          4. Works with other Android features like Backup and restore. If a user switches to a new device and restores the previously backed up data, your app will retain the user’s last preferred language, thus providing your users with a better and more seamless experience.

          Recap

          With that, most parts of the feature are covered. So let’s have a quick recap on what we discussed in today’s read.

          1. A quick read on what Per-App Language Preferences offer to multilingual users and app developers.
          2. What end users will see on their devices.
          3. How to migrate your app to the Per-App Language Preferences APIs.
          4. A few things that need to be taken care of while migrating to the APIs to ensure a better user experience.
          5. Lastly, the benefits that end users and developers can leverage from this feature.

          References

          1. Per-App Language Preferences
          2. Sample App ( Compose )
          3. Sample App ( Views )
          4. Per-App Language Preferences (YouTube Video)

          Extending support for App Engine bundled services (Module 17)

          Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

          Background

          App Engine initially launched in 2008, providing a suite of bundled services making it convenient for applications to access a database (Datastore), caching service (Memcache), independent task execution (Task Queue), Google Sign-In authentication (Users), or large "blob" storage (Blobstore), or other companion services. However, apps leveraging those services can only run their apps on App Engine.

          To increase app portability and help Google move towards its goal of having the most open cloud on the market, App Engine launched its 2nd-generation service in 2018, initially removing those legacy services. The newer platform allows developers to upgrade apps to the latest language runtimes, such as moving from Python 2 to 3 or Java 8 to 11 (and today, Java 17). One of the major drawbacks to the 1st-generation runtimes is that they're customized, proprietary, and restrictive in what you can use or can't.

          Instead, the 2nd-generation platform uses open source runtimes, meaning ability to follow standard development practices, use common/known idioms, and have fewer restrictions of 3rd-party libraries, and obviating the need to copy or "vendor" them with your code. Unfortunately, to use these newer runtimes, migrating away from App Engine services were required because while you could upgrade language releases, there was no access to bundled services, breaking apps or requiring complete rewrites, making it a showstopper for many users.

          Due to their popularity and the desire to ease the upgrade process for customers, the App Engine team restored access to most (but not all) of those services in Fall 2021. Today's Serverless Migration Station video demonstrates how to continue usage of bundled services available to Python 3 developers.

          Showing App Engine users how to use bundled services on Python 3


          Performing the upgrade

          Modernizing the typical Python 2 App Engine app looks something like this:
          1. Migrate from the webapp2 framework (not available in Python 3)
          2. Port from Python 2 to 3, preserve use of bundled services
          3. Optional migration to Cloud standalone or similar 3rd-party services

          The first step is to move to a standard Python web framework like Flask, Django, Pyramid, etc. Below is some pseudocode from Migration Module 1 demonstrating how to migrate from webapp2 to Flask:

          codeblocks for porting Python 2 sample app from webapp2 to Flask
          Step 1: Port Python 2 sample app from webapp2 to Flask

          The key changes are bolded in the above code snippets. Notice how the App Engine NDB code [the Visit class definition plus store_visit() and fetch_visits() functions] are unaffected by this web framework migration. The full webapp2 code sample can be found in the Module 0 repo folder while the completed migration to Flask sample is located in the Module 1 repo folder.

          After your app has ported frameworks, you're free to upgrade to Python 3 while preserving access to the bundled services if your app uses any. Below is pseudocode demonstrating how to upgrade the same sample app to Python 3 as well as the code changes needed to continue to use App Engine NDB:

          codeblocks for porting sample app to Python 3, preserving use of NDB bundled service
          Step 2: Port sample app to Python 3, preserving use of NDB bundled service

          The original app was designed to work under both Python 2 and 3 interpreters, so no language changes were required in this case. We added an import of the new App Engine SDK followed by the key update wrapping the WSGI object so the app can access the bundled services. As before, the key updates are bolded. Some updates to configuration are also required, and those are outlined in the documentation and the (Module 17) codelab.

          The NDB code is also left untouched in this migration. Not all of the bundled services feature such a hands-free migration, and we hope to cover some of the more complex ones ahead in Module 22. Java, PHP, and Go users have it even better, requiring fewer or no code changes at all. The Python 2 Flask sample is located in the Module 1 repo folder, and the resulting Python 3 app can be found in the Module 1b repo folder.

          The immediate benefit of step two is the ability to upgrade to a more current version of language runtime. This leaves the third step of migrating off the bundled services as optional, especially if you plan on staying on App Engine for the long-term.


          Additional options

          If you decide to migrate off the bundled services, you can do so on your own timeline. It should be a consideration should you ever want to move to modern serverless platforms such as Cloud Functions or Cloud Run, to lower-level platforms because you want more control, like GKE, our managed Kubernetes service, or Compute Engine VMs.

          Step three is also where the rest of the Serverless Migration Station content may be useful:

          *code samples and codelabs available; videos forthcoming

          As far as moving to modern serverless platforms, if you want to break apart a large App Engine app into multiple microservices, consider Cloud Functions. If your organization has added containerization as part of your software development workflow, consider Cloud Run. It's suitable for apps if you're familiar with containers and Docker, but even if you or your team don't have that experience, Cloud Buildpacks can do the heavy lifting for you. Here are the relevant migration modules to explore:


            Wrap-up

            Early App Engine users appreciate the convenience of the platform's bundled services, and after listening to user feedback, adding them back to 2nd-generation runtimes is another way we can help developers modernize their apps. Whether upgrading to newer language runtimes to stay on App Engine and continue to use its bundled services, migrating to Cloud standalone products, or shifting to other serverless platforms, the Google Cloud team aims to provide the tools to help streamline your modernization efforts.

            All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, the Cloud team is working on covering other language runtimes, so stay tuned. Today's video features a special guest to provide a teaser of what to expect for Java. For additional video content, check out the broader Serverless Expeditions series.

            Migrating from App Engine Blobstore to Cloud Storage (Module 16)

            Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

            Introduction and background

            The most recent Serverless Migration Station video demonstrated how to add use of the App Engine's Blobstore service to a sample Python 2 App Engine app, kicking off the first of a 2-part series on migrating away from Blobstore. In today's Module 16 video, we complete this journey, arriving at Cloud Storage. Moving away from proprietary App Engine services like Blobstore makes apps more portable, giving them enough flexibility to:


            Showing App Engine users how to migrate to Cloud Storage

            As described previously, a Blobstore for Python 2 dependency on webapp made the Module 15 content more straightforward to implement if it was still using webapp2. To completely modernize this app here in Module 16, the following migrations should be carried out:

            • Migrate from webapp2 (and webapp) to Flask
            • Migrate from App Engine NDB to Cloud NDB
            • Migrate from App Engine Blobstore to Cloud Storage
            • Migrate from Python 2 to Python (2 and) 3

            Performing the migrations

            Prior to modifying the application code, a variety of configuration updates need to be made. Updates applying only to Python 2 feature a "Py2" designation while those migrating to Python 3 will see "Py3" annotations.

            1. Remove the built-in Jinja2 library from app.yaml—Jinja2 already comes with Flask, so remove use of the older built-in version which may possibly conflict with the contemporary Flask version you're using. (Py2)
            2. Use of Cloud client libraries (such as those for Cloud NDB and Cloud Storage) require a pair of built-in libraries, grpcio and setuptools, so add those to app.yaml (Py2)
            3. Remove everything in app.yaml except for a valid runtime (Py3)
            4. Add Cloud NDB and Cloud Storage client libraries to requirements.txt (Py2 & Py3)
            5. Create an appengine_config.py supporting both built-in (those in app.yaml) and non built-in (those in requirements.txt) libraries used (Py2)

            The Module 15 app already migrated away from webapp2's (Django) templating system to Jinja2. This is useful when migrating to Flask because Jinja2 is Flask's default template system. Switching from App Engine NDB to Cloud NDB is fairly straightforward as the latter was designed to be mostly compatible with the original. The only change visible in this sample app is to move Datastore calls into Python with blocks.

            The most significant changes occur when moving the upload and download handlers from webapp to Cloud Storage. The video and corresponding codelab go more in-depth into the necessary changes, but in summary, these are the updates required in the main application:

            1. webapp2 is replaced by Flask. Instead of using the older built-in version of Jinja2, use the version that comes with Flask.
            2. App Engine Blobstore and NDB are replaced by Cloud NDB and Cloud Storage, respectively.
            3. The webapp Blobstore handler functionality is replaced by a combination of the io standard library module plus components from Flask and Werkzeug. Furthermore, the handler classes and methods are replaced by Flask functions.
            4. The main handler class and corresponding GET and POST methods are all replaced by a single Flask function.

            The results

            With all the changes implemented, the original Module 15 app still operates identically in Module 16, starting with a form requesting a visit artifact followed by the most recents visits page:
            The sample app's artifact prompt page

            The sample app's most recent visits page.

            The only difference is that four migrations have been completed where all of the "infrastructure" is now taken care of by non-App Engine legacy services. Furthermore, the Module 16 app could be either a Python 2 or 3 app. As far as the end-user is concerned, "nothing happened."

            Migrating sample app from App Engine Blobstore to Cloud Storage

            Wrap-up

            Module 16 featured four different migrations, modernizing the Module 15 app from using App Engine legacy services like NDB and Blobstore to Cloud NDB and Cloud Storage, respectively. While we recommend users move to the latest offerings from Google Cloud, migrating from Blobstore to Cloud Storage isn't required, and should you opt to do so, can do it on your own timeline. In addition to today's video, be sure to check out the Module 16 codelab which leads you step-by-step through the migrations discussed.

            In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (that have a 1st generation runtime), meaning you are no longer required to migrate to Cloud Storage when porting your app to Python 3. You can continue using Blobstore in your Python 3 app so long as you retrofit the code to access bundled services from next-generation runtimes.

            If you're using other App Engine legacy services be sure to check out the other Migration Modules in this series. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. While our content initially focuses on Python users, the Cloud team is working on covering other language runtimes, so stay tuned. For additional video content, check out our broader Serverless Expeditions series.

            How can App Engine users take advantage of Cloud Functions?

            Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

            Introduction

            Recently, we discussed containerizing App Engine apps for Cloud Run, with or without Docker. But what about Cloud Functions… can App Engine users take advantage of that platform somehow? Back in the day, App Engine was always the right decision, because it was the only option. With Cloud Functions and Cloud Run joining in the serverless product suite, that's no longer the case.

            Back when App Engine was the only choice, it was selected to host small, single-function apps. Yes, when it was the only option. Other developers have created huge monolithic apps for App Engine as well… because it was also the only option. Fast forward to today where code follows more service-oriented or event-driven architectures. Small apps can be moved to Cloud Functions to simplify the code and deployments while large apps could be split into smaller components, each running on Cloud Functions.

            Refactoring App Engine apps for Cloud Functions

            Small, single-function apps can be seen as a microservice, an API endpoint "that does something," or serve some utility likely called as a result of some event in a larger multi-tiered application, say to update a database row or send a customer email message. App Engine apps require some kind web framework and routing mechanism while Cloud Function equivalents can be freed from much of those requirements. Refactoring these types of App Engine apps for Cloud Functions will like require less overhead, helps ease maintenance, and allow for common components to be shared across applications.

            Large, monolithic applications are often made up of multiple pieces of functionality bundled together in one big package, such as requisitioning a new piece of equipment, opening a customer order, authenticating users, processing payments, performing administrative tasks, and so on. By breaking this monolith up into multiple microservices into individual functions, each component can then be reused in other apps, maintenance is eased because software bugs will identify code closer to their root origins, and developers won't step on each others' toes.

            Migration to Cloud Functions

            In this latest episode of Serverless Migration Station, a Serverless Expeditions mini-series focused on modernizing serverless apps, we take a closer look at this product crossover, covering how to migrate App Engine code to Cloud Functions. There are several steps you need to take to prepare your code for Cloud Functions:

            • Divest from legacy App Engine "bundled services," e.g., Datastore, Taskqueue, Memcache, Blobstore, etc.
            • Cloud Functions supports modern runtimes; upgrade to Python 3, Java 11, or PHP 7
            • If your app is a monolith, break it up into multiple independent functions. (You can also keep a monolith together and containerize it for Cloud Run as an alternative.)
            • Make appropriate application updates to support Cloud Functions

              The first three bullets are outside the scope of this video and its codelab, so we'll focus on the last one. The changes needed for your app include the following:

              1. Remove unneeded and/or unsupported configuration
              2. Remove use of the web framework and supporting routing code
              3. For each of your functions, assign an appropriate name and install the request object it will receive when it is called.

              Regarding the last point, note that you can have multiple "endpoints" coming into a single function which processes the request path, calling other functions to handle those routes. If you have many functions in your app, separate functions for every endpoint becomes unwieldy; if large enough, your app may be more suited for Cloud Run. The sample app in this video and corresponding code sample only has one function, so having a single endpoint for that function works perfectly fine here.

              This migration series focuses on our earliest users, starting with Python 2. Regarding the first point, the app.yaml file is deleted. Next, almost all Flask resources are removed except for the template renderer (the app still needs to output the same HTML as the original App Engine app). All app routes are removed, and there's no instantiation of the Flask app object. Finally for the last step, the main function is renamed more appropriately to visitme() along with a request object parameter.

              This "migration module" starts with the (Python 3 version of the) Module 2 sample app, applies the steps above, and arrives at the migrated Module 11 app. Implementing those required changes is illustrated by this code "diff:"

              Migration of sample app to Cloud Functions

              Next steps

              If you're interested in trying this migration on your own, feel free to try the corresponding codelab which leads you step-by-step through this exercise and use the video for additional guidance.

              All migration modules, their videos (when published), codelab tutorials, START and FINISH code, etc., can be found in the migration repo. We hope to also one day cover other legacy runtimes like Java 8 as well as content for the next-generation Cloud Functions service, so stay tuned. If you're curious whether it's possible to write apps that can run on App Engine, Cloud Functions, or Cloud Run with no code changes at all, the answer is yes. Hope this content is useful for your consideration when modernizing your own serverless applications!

            An easier way to move your App Engine apps to Cloud Run

            Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

            Blue header

            An easier yet still optional migration

            In the previous episode of the Serverless Migration Station video series, developers learned how to containerize their App Engine code for Cloud Run using Docker. While Docker has gained popularity over the past decade, not everyone has containers integrated into their daily development workflow, and some prefer "containerless" solutions but know that containers can be beneficial. Well today's video is just for you, showing how you can still get your apps onto Cloud Run, even If you don't have much experience with Docker, containers, nor Dockerfiles.

            App Engine isn't going away as Google has expressed long-term support for legacy runtimes on the platform, so those who prefer source-based deployments can stay where they are so this is an optional migration. Moving to Cloud Run is for those who want to explicitly move to containerization.

            Migrating to Cloud Run with Cloud Buildpacks video

            So how can apps be containerized without Docker? The answer is buildpacks, an open-source technology that makes it fast and easy for you to create secure, production-ready container images from source code, without a Dockerfile. Google Cloud Buildpacks adheres to the buildpacks open specification and allows users to create images that run on all GCP container platforms: Cloud Run (fully-managed), Anthos, and Google Kubernetes Engine (GKE). If you want to containerize your apps while staying focused on building your solutions and not how to create or maintain Dockerfiles, Cloud Buildpacks is for you.

            In the last video, we showed developers how to containerize a Python 2 Cloud NDB app as well as a Python 3 Cloud Datastore app. We targeted those specific implementations because Python 2 users are more likely to be using App Engine's ndb or Cloud NDB to connect with their app's Datastore while Python 3 developers are most likely using Cloud Datastore. Cloud Buildpacks do not support Python 2, so today we're targeting a slightly different audience: Python 2 developers who have migrated from App Engine ndb to Cloud NDB and who have ported their apps to modern Python 3 but now want to containerize them for Cloud Run.

            Developers familiar with App Engine know that a default HTTP server is provided by default and started automatically, however if special launch instructions are needed, users can add an entrypoint directive in their app.yaml files, as illustrated below. When those App Engine apps are containerized for Cloud Run, developers must bundle their own server and provide startup instructions, the purpose of the ENTRYPOINT directive in the Dockerfile, also shown below.

            Starting your web server with App Engine (app.yaml) and Cloud Run with Docker (Dockerfile) or Buildpacks (Procfile)

            Starting your web server with App Engine (app.yaml) and Cloud Run with Docker (Dockerfile) or Buildpacks (Procfile)

            In this migration, there is no Dockerfile. While Cloud Buildpacks does the heavy-lifting, determining how to package your app into a container, it still needs to be told how to start your service. This is exactly what a Procfile is for, represented by the last file in the image above. As specified, your web server will be launched in the same way as in app.yaml and the Dockerfile above; these config files are deliberately juxtaposed to expose their similarities.

            Other than this swapping of configuration files and the expected lack of a .dockerignore file, the Python 3 Cloud NDB app containerized for Cloud Run is nearly identical to the Python 3 Cloud NDB App Engine app we started with. Cloud Run's build-and-deploy command (gcloud run deploy) will use a Dockerfile if present but otherwise selects Cloud Buildpacks to build and deploy the container image. The user experience is the same, only without the time and challenges required to maintain and debug a Dockerfile.

            Get started now

            If you're considering containerizing your App Engine apps without having to know much about containers or Docker, we recommend you try this migration on a sample app like ours before considering it for yours. A corresponding codelab leading you step-by-step through this exercise is provided in addition to the video which you can use for guidance.

            All migration modules, their videos (when available), codelab tutorials, and source code, can be found in the migration repo. While our content initially focuses on Python users, we hope to one day also cover other legacy runtimes so stay tuned. Containerization may seem foreboding, but the goal is for Cloud Buildpacks and migration resources like this to aid you in your quest to modernize your serverless apps!

            Containerizing Google App Engine apps for Cloud Run

            Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud

            Google App Engine header

            An optional migration

            Serverless Migration Station is a video mini-series from Serverless Expeditions focused on helping developers modernize their applications running on a serverless compute platform from Google Cloud. Previous episodes demonstrated how to migrate away from the older, legacy App Engine (standard environment) services to newer Google Cloud standalone equivalents like Cloud Datastore. Today's product crossover episode differs slightly from that by migrating away from App Engine altogether, containerizing those apps for Cloud Run.

            There's little question the industry has been moving towards containerization as an application deployment mechanism over the past decade. However, Docker and use of containers weren't available to early App Engine developers until its flexible environment became available years later. Fast forward to today where developers have many more options to choose from, from an increasingly open Google Cloud. Google has expressed long-term support for App Engine, and users do not need to containerize their apps, so this is an optional migration. It is primarily for those who have decided to add containerization to their application deployment strategy and want to explicitly migrate to Cloud Run.

            If you're thinking about app containerization, the video covers some of the key reasons why you would consider it: you're not subject to traditional serverless restrictions like development language or use of binaries (flexibility); if your code, dependencies, and container build & deploy steps haven't changed, you can recreate the same image with confidence (reproducibility); your application can be deployed elsewhere or be rolled back to a previous working image if necessary (reusable); and you have plenty more options on where to host your app (portability).

            Migration and containerization

            Legacy App Engine services are available through a set of proprietary, bundled APIs. As you can surmise, those services are not available on Cloud Run. So if you want to containerize your app for Cloud Run, it must be "ready to go," meaning it has migrated to either Google Cloud standalone equivalents or other third-party alternatives. For example, in a recent episode, we demonstrated how to migrate from App Engine ndb to Cloud NDB for Datastore access.

            While we've recently begun to produce videos for such migrations, developers can already access code samples and codelab tutorials leading them through a variety of migrations. In today's video, we have both Python 2 and 3 sample apps that have divested from legacy services, thus ready to containerize for Cloud Run. Python 2 App Engine apps accessing Datastore are most likely to be using Cloud NDB whereas it would be Cloud Datastore for Python 3 users, so this is the starting point for this migration.

            Because we're "only" switching execution platforms, there are no changes at all to the application code itself. This entire migration is completely based on changing the apps' configurations from App Engine to Cloud Run. In particular, App Engine artifacts such as app.yaml, appengine_config.py, and the lib folder are not used in Cloud Run and will be removed. A Dockerfile will be implemented to build your container. Apps with more complex configurations in their app.yaml files will likely need an equivalent service.yaml file for Cloud Run — if so, you'll find this app.yaml to service.yaml conversion tool handy. Following best practices means there'll also be a .dockerignore file.

            App Engine and Cloud Functions are sourced-based where Google Cloud automatically provides a default HTTP server like gunicorn. Cloud Run is a bit more "DIY" because users have to provide a container image, meaning bundling our own server. In this case, we'll pick gunicorn explicitly, adding it to the top of the existing requirements.txt required packages file(s), as you can see in the screenshot below. Also illustrated is the Dockerfile where gunicorn is started to serve your app as the final step. The only differences for the Python 2 equivalent Dockerfile are: a) require the Cloud NDB package (google-cloud-ndb) instead of Cloud Datastore, and b) start with a Python 2 base image.

            Image of The Python 3 requirements.txt and Dockerfile

            The Python 3 requirements.txt and Dockerfile

            Next steps

            To walk developers through migrations, we always "START" with a working app then make the necessary updates that culminate in a working "FINISH" app. For this migration, the Python 2 sample app STARTs with the Module 2a code and FINISHes with the Module 4a code. Similarly, the Python 3 app STARTs with the Module 3b code and FINISHes with the Module 4b code. This way, if something goes wrong during your migration, you can always rollback to START, or compare your solution with our FINISH. If you are considering this migration for your own applications, we recommend you try it on a sample app like ours before considering it for yours. A corresponding codelab leading you step-by-step through this exercise is provided in addition to the video which you can use for guidance.

            All migration modules, their videos (when published), codelab tutorials, START and FINISH code, etc., can be found in the migration repo. We hope to also one day cover other legacy runtimes like Java 8 so stay tuned. We'll continue with our journey from App Engine to Cloud Run ahead in Module 5 but will do so without explicit knowledge of containers, Docker, or Dockerfiles. Modernizing your development workflow to using containers and best practices like crafting a CI/CD pipeline isn't always straightforward; we hope content like this helps you progress in that direction!