Tag Archives: Google Cloud Platform

Machine Learning GDEs: Q1 2021 highlights, projects and achievements

Posted by HyeJung Lee and MJ You, Google ML Ecosystem Community Managers. Reviewed by Soonson Kwon, Developer Relations Program Manager.

Google Developers Experts is a community of passionate developers who love to share their knowledge with others. Many of them specialize in Machine Learning (ML). Despite many unexpected changes over the last months and reduced opportunities for various in person activities during the ongoing pandemic, their enthusiasm did not stop.

Here are some highlights of the ML GDE’s hard work during the Q1 2021 which contributed to the global ML ecosystem.

ML GDE YouTube channel

ML GDE YouTube page

With the initiative and lead of US-based GDE Margaret Maynard-Reid, we launched the ML GDEs YouTube channel. It is a great way for GDEs to reach global audiences, collaborate as a community, create unique content and promote each other's work. It will contain all kinds of ML related topics: talks on technical topics, tutorials, interviews with another (ML) GDE, a Googler or anyone in the ML community etc. Many videos have already been uploaded, including: ML GDE’s intro from all over the world, tips for TensorFlow & GCP Certification and how to use Google Cloud Platform etc. Subscribe to the channel now!!

TensorFlow Everywhere

TensorFlow Everywhere logo

17 ML GDEs presented at TensorFlow Everywhere (a global community-led event series for TensorFlow and Machine Learning enthusiasts and developers around the world) hosted by local TensorFlow user groups. You can watch the recorded sessions in the TensorFlow Everywhere playlist on the ML GDE Youtube channel. Most of the sessions cover new features in Tensorflow.

International Women’s Day

Many ML GDEs participated in activities to celebrate International Women’s Day (March 8th). GDE Ruqiya Bin Safi (based in Saudi Arabia) cooperated with WTM Saudi Arabia to organize “Socialthon” - social development hackathons and gave a talk “Successful Experiences in Social Development", which reached 77K viervers live and hit 10K replays. India-based GDE Charmi Chokshi participated in GirlScript's International Women's Day event and gave a talk: “Women In Tech and How we can help the underrepresented in the challenging world”. If you’re looking for more inspiring materials, check out the “Women in AI” playlist on our ML GDE YouTube channel!

Mentoring

ML GDEs are also very active in mentoring community developers, students in the Google Developer Student Clubs and startups in the Google for Startups Accelerator program. Among many, GDE Arnaldo Gualberto (Brazil) conducted mentorship sessions for startups in the Google Fast Track program, discussing how to solve challanges using Machine Learning/Deep Learning with TensorFlow.

TensorFlow

Practical Adversarial Robustness in Deep Learning: Problems and Solutions
ML using TF cookbook and ML for Dummies book

Meanwhile in Europe, GDEs Alexia Audevart (based in France) and Luca Massaron (based in Italy) released “Machine Learning using TensorFlow Cookbook”. It provides simple and effective ideas to successfully use TensorFlow 2.x in computer vision, NLP and tabular data projects. Additionally, Luca published the second edition of the Machine Learning For Dummies book, first published in 2015. Her latest edition is enhanced with product updates and the principal is a larger share of pages devoted to discussion of Deep Learning and TensorFlow / Keras usage.

YouTube video screenshot

On top of her women-in-tech related activities, Ruqiya Bin Safi is also running a “Welcome to Deep Learning Course and Orientation” monthly workshop throughout 2021. The course aims to help participants gain foundational knowledge of deep learning algorithms and get practical experience in building neural networks in TensorFlow.

TensorFlow Project showcase

Nepal-based GDE Kshitiz Rimal gave a talk “TensorFlow Project Showcase: Cash Recognition for Visually Impaired" on his project which uses TensorFlow, Google Cloud AutoML and edge computing technologies to create a solution for the visually impaired community in Nepal.

Screenshot of TF Everywhere NA talk

On the other side of the world, in Canada, GDE Tanmay Bakshi presented a talk “Machine Learning-powered Pipelines to Augment Human Specialists” during TensorFlow Everywhere NA. It covered the world of NLP through Deep Learning, how it's historically been done, the Transformer revolution, and how using the TensorFlow & Keras to implement use cases ranging from small-scale name generation to large-scale Amazon review quality ranking.

Google Cloud Platform

Google Cloud Platform YouTube playlist screenshot

We have been equally busy on the GCP side as well. In the US, GDE Srivatsan Srinivasan created a series of videos called “Artificial Intelligence on Google Cloud Platform”, with one of the episodes, "Google Cloud Products and Professional Machine Learning Engineer Certification Deep Dive", getting over 3,000 views.

ML Analysis Pipeline

Korean GDE Chansung Park contributed to TensorFlow User Group Korea with his “Machine Learning Pipeline (CI/CD for ML Products in GCP)” analysis, focused on about machine learning pipeline in Google Cloud Platform.

Analytics dashboard

Last but not least, GDE Gad Benram based in Israel wrote an article on “Seven Tips for Forecasting Cloud Costs”, where he explains how to build and deploy ML models for time series forecasting with Google Cloud Run. It is linked with his solution of building a cloud-spend control system that helps users more-easily analyze their cloud costs.

If you want to know more about the Google Experts community and all their global open-source ML contributions, visit the GDE Directory and connect with GDEs on Twitter and LinkedIn. You can also meet them virtually on the ML GDE’s YouTube Channel!

Modernizing Oracle operations with Kubernetes and El Carro

Google Cloud is releasing El Carro, an open source tool to help you transform and modernize your Oracle database operations. El Carro implements the Kubernetes operator pattern to deliver automation for provisioning and ongoing operations like backups, patching, and high availability for databases running in hybrid and multi-cloud environments. And it does so using the same declarative syntax that DevOps teams are using to manage applications. With El Carro, users can choose to modernize and transform their database operations in place and benefit from a consistent management experience and hybrid and multi-cloud portability. Released under the Apache License 2.0, you are free to use El Carro in any Kubernetes environment—you are in control.

Containers and Kubernetes deliver portability on standardized infrastructure, and today Oracle supports databases running in containers; they’ve also released container build files and images and helm charts to simplify provisioning. What is missing for the next level of integration is support for lifecycle operations and an extension of the Kubernetes API to the primitives needed for database management.

In addition, fully managed or autonomous services for Oracle may not make available all the required features, such as Active Data Guard, Multitenant, and In-Memory, parameters/flags, versions, and patch levels. DBAs also find themselves locked out of many roles, including sysadmin and root. These restrictions make many cloud architects fall back to lift and shift Oracle databases onto infrastructure as a service offerings and miss out on opportunities to modernize and transform database operations. And with transactional databases growing in number and criticality, organizations are struggling to deliver innovation and modernization. Engineers are already busy keeping up with sprawl and mundane operational tasks while adhering to strict change management processes.

How do we solve this database operations gap?

El Carro solves this. It is built with scalability in mind, using the same container orchestration infrastructure, Kubernetes, that powers many businesses and is a top choice for modern architectures. Its open API allows you to manage your database configurations as declarative code, enabling CI/CD or Gitops workflows for auditability and control mechanisms. El Carro automates many database lifecycle operations, like backups, replication, and patching. And, when it distributes databases on the nodes of a cluster, it is aware of the priority and resource requirements of each database to optimize tight packing while respecting quality of service. Lastly, it helps DBAs by delivering automation without restrictions and leaving DBAs in full control over their systems. You can choose to let the operator drive for you, but you can also take over the steering wheel yourself at any time.

Because Kubernetes is now the standard for portable infrastructure automation and orchestration, engineers appreciate how Kubernetes abstracts complex problems into manageable infrastructure as code. Kubernetes can scale from small projects to large projects that support the infrastructure that powers Google products and services for billions of users around the world. Moreover, Google pioneers the next generation of infrastructure as code that we refer to as Configuration as Data to declaratively establish a contract between developer intent and the runtime operation. According to the Cloud Native Survey 2020, two-thirds of respondents were either already running stateful workloads in production or were considering doing so within the next 12 months. We expect that datastores are going to drive the next wave of enterprise Kubernetes adoption.

A number of open source operators for databases, such as PostgreSQL, MySQL, and many others, have been released, are actively maintained by the community, and are popular among developers and architects looking for a hands-off approach to manage databases with their applications. El Carro extends the list of database operators to include Oracle.

What are we building with El Carro for Oracle databases?

The operator pattern emerged in late 2016 as an extension of the Kubernetes API and control loop aimed at automating more complicated and application-specific tasks that are beyond the native Kubernetes objects.

El Carro implements a custom resource definition (CRD), which is tailored to database management. Users set and change attributes of the custom resource using the Kubernetes API the same way they do for built-in objects such as pods, deployments, or services. The El Carro controller observes changes to the CRs and compares the declared state with the current reality in the cluster, then makes the necessary changes. Those changes could either affect the Kubernetes resources used by the database such as persistent volumes or the pod itself, or may result in issuing calls via SQL or command line tools to the database to create and modify users or other database objects.

Here’s a look at how this works:
El Carro Architecture
El Carro Architecture

The diagram above shows how the major components of a database managed by the El Carro Operator interact with each other. The controller monitors the CRD for any changes made by admins. It creates and manages the cluster resources that make up the actual database deployment: persistent volumes for filesystems and data, a pod to run containers with the actual database, and a daemon that allows the controller to securely run SQL commands on the database. And lastly, a service makes sqlnet connections available to applications and end users that can either run in the same Kubernetes cluster or outside of it.

At release time, the El Carro Operator can provision Oracle databases of 12c Enterprise Edition and 18c Express Edition. It manages instance parameters, pluggable databases, and users. You can take and restore backups either using rman or storage snapshots, and we are working to add additional features.

How to get involved with El Carro?

In the development process, we collaborated with users and partners in the Oracle community to help us validate the approach. "Pythian has helped Oracle users to automate and optimize the operations of their mission-critical systems for over 20 years,” says Simon Pane, principal consultant at Pythian. “We are excited about the possibilities that El Carro brings to users on their cloud modernization journeys. We are proud to work with the community on a vision for the future of database management.".

Sean Scott covers Docker for databases on his blog oraclesean.com, and says: "There are many benefits to running Oracle databases in containers. Adding Kubernetes orchestration introduces new opportunities to bring the DevOps and Oracle communities together."

You can try out El Carro today. Follow the quick start guide and try out provisioning of instances, databases, users. Import data via Data Pump, manage instance parameters, choose between different methods for backups, and try out a restore. Have a look at how we integrate with external logging and monitoring solutions. Reach out via our Google group and leave feedback for what features you would like to see next, or even create your own patch and pull request on GitHub.

By Bjoern Rost - Product Manager and Boris Dali - Team Lead, Engineering

Google Developer Group Spotlight: A conversation with Cloud Architect, Ilias Papachristos

Posted by Jennifer Kohl, Global Program Manager, Google Developer Communities

The Google Developer Groups Spotlight series interviews inspiring leaders of community meetup groups around the world. Our goal is to learn more about what developers are working on, how they’ve grown their skills with the Google Developer Group community, and what tips they might have for us all.

We recently spoke with Ilias Papachristos, Google Developer Group Cloud Thessaloniki Lead in Greece. Check out our conversation with Ilias on Cloud architecture, reading official documentation, and suggested resources to help developers grow professionally.

Tell us a little about yourself?

I’m a family man, ex-army helicopter pilot, Kendo sensei, beta tester at Coursera, Lead of the Google Developer Group Cloud Thessaloniki community, Google Cloud Professional Architect, and a Cloud Board Moderator on the Google Developers Community Leads Platform (CLP).

I love outdoor activities, reading books, listening to music, and cooking for my family and friends!

Can you explain your work in Cloud technologies?

Over my career, I have used Compute Engine for an e-shop, AutoML Tables for an HR company, and have architected the migration of a company in Mumbai. Now I’m consulting for a company on two of their projects: one that uses Cloud Run and another that uses Kubernetes.

Both of them have Cloud SQL and the Kubernetes project will use the AI Platform. We might even end up using Dataflow with BigQuery for the streaming and Scheduler or Manager, but I’m still working out the details.

I love the chance to share knowledge with the developer community. Many days, I open my PC, read the official Google Cloud blog, and share interesting articles on the CLP Cloud Board and GDG Cloud Thessaloniki’s social media accounts. Then, I check Google Cloud’s Medium publication for extra articles. Read, comment, share, repeat!

How did the Google Developer Group community help your Cloud career?

My overall knowledge of Google Cloud has to do with my involvement with Google Developer Groups. It is not just one thing. It’s about everything! At the first European GDG Leads Summit, I met so many people who were sharing their knowledge and offering their help. For a newbie like me it was and still is something that I keep in my heart as a treasure

I’ve also received so many informative lessons on public speaking from Google Developer Group and Google Developer Student Club Leads. They always motivate me to continue talking about the things I love!

What has been the most inspiring part of being a part of your local Google Developer Group?

Collaboration with the rest of the DevFest Hellas Team! For this event, I was a part of a small group of 12 organizers, all of whom never had hosted a large meetup before. With the help of Google Developer Groups, we had so much fun while creating a successful DevFest learning program for 360 people.

What are some technical resources you have found the most helpful for your professional development?

Besides all of the amazing tricks and tips you can learn from the Google Cloud training team and courses on the official YouTube channel, I had the chance to hear a talk by Wietse Venema on Cloud Run. I also have learned so much about AI from Dale Markovitz’s videos on Applied AI. And of course, I can’t leave out Priyanka Vergadia’s posts, articles, and comic-videos!

Official documentation has also been a super important part of my career. Here are five links that I am using right now as an Architect:

  1. Google Cloud Samples
  2. Cloud Architecture Center
  3. Solve with Google Cloud
  4. Google Cloud Solutions
  5. 13 sample architectures to kickstart your Google Cloud journey

How did you become a Google Developer Group Lead?

I am a member of the Digital Analytics community in Thessaloniki, Greece. Their organizer asked me to write articles to start motivating young people. I translated one of the blogs into English and published it on Medium. The Lead of GDG Thessaloniki read them and asked me to become a facilitator for a Cloud Study Jams (CSJ) workshop. I accepted and then traveled to Athens to train three people so that they could also become CSJ facilitators. At the end of the CSJ, I was asked if I wanted to lead a Google Developer Group chapter. I agreed. Maria Encinar and Katharina Lindenthal interviewed me, and I got it!

What would be one piece of advice you have for someone looking to learn more about a specific technology?

Learning has to be an amusing and fun process. And that’s how it’s done with Google Developer Groups all over the world. Join mine, here. It’s the best one. (Wink, wink.)

Want to start growing your career and coding knowledge with developers like Ilias? Then join a Google Developer Group near you, here.

Modernizing your Google App Engine applications

Posted by Wesley Chun, Developer Advocate, Google Cloud

Modernizing your Google App Engine applications header

Next generation service

Since its initial launch in 2008 as the first product from Google Cloud, Google App Engine, our fully-managed serverless app-hosting platform, has been used by many developers worldwide. Since then, the product team has continued to innovate on the platform: introducing new services, extending quotas, supporting new languages, and adding a Flexible environment to support more runtimes, including the ability to serve containerized applications.

With many original App Engine services maturing to become their own standalone Cloud products along with users' desire for a more open cloud, the next generation App Engine launched in 2018 without those bundled proprietary services, but coupled with desired language support such as Python 3 and PHP 7 as well as introducing Node.js 8. As a result, users have more options, and their apps are more portable.

With the sunset of Python 2, Java 8, PHP 5, and Go 1.11, by their respective communities, Google Cloud has assured users by expressing continued long-term support of these legacy runtimes, including maintaining the Python 2 runtime. So while there is no requirement for users to migrate, developers themselves are expressing interest in updating their applications to the latest language releases.

Google Cloud has created a set of migration guides for users modernizing from Python 2 to 3, Java 8 to 11, PHP 5 to 7, and Go 1.11 to 1.12+ as well as a summary of what is available in both first and second generation runtimes. However, moving from bundled to unbundled services may not be intuitive to developers, so today we're introducing additional resources to help users in this endeavor: App Engine "migration modules" with hands-on "codelab" tutorials and code examples, starting with Python.

Migration modules

Each module represents a single modernization technique. Some are strongly recommended, others less so, and, at the other end of the spectrum, some are quite optional. We will guide you as far as which ones are more important. Similarly, there's no real order of modules to look at since it depends on which bundled services your apps use. Yes, some modules must be completed before others, but again, you'll be guided as far as "what's next."

More specifically, modules focus on the code changes that need to be implemented, not changes in new programming language releases as those are not within the domain of Google products. The purpose of these modules is to help reduce the friction developers may encounter when adapting their apps for the next-generation platform.

Central to the migration modules are the codelabs: free, online, self-paced, hands-on tutorials. The purpose of Google codelabs is to teach developers one new skill while giving them hands-on experience, and there are codelabs just for Google Cloud users. The migration codelabs are no exception, teaching developers one specific migration technique.

Developers following the tutorials will make the appropriate updates on a sample app, giving them the "muscle memory" needed to do the same (or similar) with their applications. Each codelab begins with an initial baseline app ("START"), leads users through the necessary steps, then concludes with an ending code repo ("FINISH") they can compare against their completed effort. Here are some of the initial modules being announced today:

  • Web framework migration from webapp2 to Flask
  • Updating from App Engine ndb to Google Cloud NDB client libraries for Datastore access
  • Upgrading from the Google Cloud NDB to Cloud Datastore client libraries
  • Moving from App Engine taskqueue to Google Cloud Tasks
  • Containerizing App Engine applications to execute on Cloud Run

Examples

What should you expect from the migration codelabs? Let's preview a pair, starting with the web framework: below is the main driver for a simple webapp2-based "guestbook" app registering website visits as Datastore entities:

class MainHandler(webapp2.RequestHandler):
'main application (GET) handler'
def get(self):
store_visit(self.request.remote_addr, self.request.user_agent)
visits = fetch_visits(LIMIT)
tmpl = os.path.join(os.path.dirname(__file__), 'index.html')
self.response.out.write(template.render(tmpl, {'visits': visits}))

A "visit" consists of a request's IP address and user agent. After visit registration, the app queries for the latest LIMIT visits to display to the end-user via the app's HTML template. The tutorial leads developers a migration to Flask, a web framework with broader support in the Python community. An Flask equivalent app will use decorated functions rather than webapp2's object model:

@app.route('/')
def root():
'main application (GET) handler'
store_visit(request.remote_addr, request.user_agent)
visits = fetch_visits(LIMIT)
return render_template('index.html', visits=visits)

The framework codelab walks users through this and other required code changes in its sample app. Since Flask is more broadly used, this makes your apps more portable.

The second example pertains to Datastore access. Whether you're using App Engine's ndb or the Cloud NDB client libraries, the code to query the Datastore for the most recent limit visits may look like this:

def fetch_visits(limit):
'get most recent visits'
query = Visit.query()
visits = query.order(-Visit.timestamp).fetch(limit)
return (v.to_dict() for v in visits)

If you decide to switch to the Cloud Datastore client library, that code would be converted to:

def fetch_visits(limit):
'get most recent visits'
query = DS_CLIENT.query(kind='Visit')
query.order = ['-timestamp']
return query.fetch(limit=limit)

The query styles are similar but different. While the sample apps are just that, samples, giving you this kind of hands-on experience is useful when planning your own application upgrades. The goal of the migration modules is to help you separate moving to the next-generation service and making programming language updates so as to avoid doing both sets of changes simultaneously.

As mentioned above, some migrations are more optional than others. For example, moving away from the App Engine bundled ndb library to Cloud NDB is strongly recommended, but because Cloud NDB is available for both Python 2 and 3, it's not necessary for users to migrate further to Cloud Datastore nor Cloud Firestore unless they have specific reasons to do so. Moving to unbundled services is the primary step to giving users more flexibility, choices, and ultimately, makes their apps more portable.

Next steps

For those who are interested in modernizing their apps, a complete table describing each module and links to corresponding codelabs and expected START and FINISH code samples can be found in the migration module repository. We are also working on video content based on these migration modules as well as producing similar content for Java, so stay tuned.

In addition to the migration modules, our team has also setup a separate repo to support community-sourced migration samples. We hope you find all these resources helpful in your quest to modernize your App Engine apps!

India’s Google Developer Groups meet up to ace their Google Cloud Certifications

Posted by Biswajeet Mallik, Program Manager, Google Developers India.

Image from Cloud Community Days India

Earlier this year, ten Google Developer Groups in India came together to host Google Cloud Community Days India, a two day event helping developers study for their upcoming Cloud Certification exams. To address the rising demand for professional certifications, the virtual event hosted over 63,000 developers, covered four main exam areas, and welcomed nine speakers. This was the second edition to the event series which started in 2019 in India.

By providing expert learning materials and mentorship, the event uniquely prepared developers for the Associate Cloud Engineer, Professional Data Engineer, Professional Cloud Machine Learning Engineer, and Professional Cloud Architect exams. Learn more below.

Acing the four key certifications

The Cloud Community Days event focused on helping developers study for four milestone certifications, tailored to engineers at four different stages of their career. The goal: help Google Developer Group members obtain the right credentials to improve their job prospects.

The event broke participants into breakout sessions based on which exam they were preparing to take. Since the certifications targeted professionals of all skill levels, study groups ranged from early career associates to late career executives. The learning groups were organized around the following certifications:

  1. Associate Cloud Engineer:

    This learning session was created to help early career developers complete the first stepping stone exam. In particular, learning materials and speakers were curated to guide participants who had no prior experience, or very little, working on the Google Cloud Platform.

    Workshops were mainly dedicated to assisting programmers who were familiar with building different applications but wished to show employers that they could deploy them on Google Cloud Platform.

    Watch more from: Day 1, here. And day 2, here.

  2. Professional Data Engineers:

    The next group brought together were data practitioners with special interests in data visualization and decision making. Workshops and learning activities helped these developers hone their large scale data and data driven decision making abilities.

    Improving these skills are essential for passing the Professional Data Engineers certification and growing a programmer’s early career.

    Watch more from: Day 1, here. And day 2, here.

  3. Professional Cloud Machine Learning Engineer:

    For these sessions, the Google Developer Group Cloud community paired experienced programmers with a significant interest in ML to form their study groups. The main driver in these learning activities was to help seasoned developers gain a deeper understanding of how to utilize Google Cloud ML services.

    With significant emphasis being placed on machine learning in the ecosystem right now, Google Developer Group community leaders felt this certification could help developers make the leap into new leadership roles.

    Watch more from: Day 1, here. And day 2, here.

  4. Professional Cloud Architect:

    Lastly, this event paired experienced Cloud executives and professionals working in leading capacities for their organizations. For these sessions, speakers and activities had a specific scope: help high level professions be at the forefront of Google Cloud Platforms innovative capabilities.

    Specifically, the Professional Cloud Architect Certification was created to help senior software engineers better design, scale and develop highly secure and robust applications.

    Day 1, here. And day 2, here.

Reactions from the community

Overall, the community put together these resources to help developers feel more confident in their abilities, obtain tangible credentials, and in turn increase access to better job opportunities. As two participants recalled the event,

“The session on Qwiklabs was so helpful, and taught me how to anticipate problems and then solve them. Cloud Community Days inspired me to take the next step with DevOps and Google Cloud.”

“This was the first time I attended the Google Developer Group event! It is an awesome package for learning in one place. All the fun activities were engaging and the panelist discussion was also very insightful. I feel proud to be a part of this grand GDG event.”

Start learning with Google Developer Groups

With Google Developer Groups, find a space to learn alongside a group of curious developers, all coming together to advance their careers from withinside a caring community of peers.

Want to know more about what Cloud Community days were like? Then watch their live recording below.

Ready to find a community event near you? Then get started at gdg.community.dev

Announcing gRPC Kotlin 1.0 for Android and Cloud

Posted by Louis Wasserman, Software Engineer and James Ward, Developer Advocate

Kotlin is now the fourth "most loved" programming language with millions of developers using it for Android, server-side / cloud backends, and various other target runtimes. At Google, we've been building more of our apps and backends with Kotlin to take advantage of its expressiveness, safety, and excellent support for writing asynchronous code with coroutines.

Since everything in Google runs on top of gRPC, we needed an idiomatic way to do gRPC with Kotlin. Back in April 2020 we announced the open sourcing of gRPC Kotlin, something we'd originally built for ourselves. Since then we've seen over 30,000 downloads and usage in Android and Cloud. The community and our engineers have been working hard polishing docs, squashing bugs, and making improvements to the project; culminating in the shiny new 1.0 release! Dive right in with the gRPC Kotlin Quickstart!

For those new to gRPC & Kotlin let's do a quick runthrough of some of the awesomeness. gRPC builds on Protocol Buffers, aka "protos" (language agnostic & high performance data interchange) and adds the network protocol for efficiently communicating with protos. From a proto definition the servers, clients, and data transfer objects can all be generated. Here is a simple gRPC proto:

message HelloRequest {
string name = 1;
}

message HelloReply {
string message = 1;
}

service Greeter {
rpc SayHello (HelloRequest) returns (HelloReply) {}
}

In a Kotlin project you can then define the implementation of the Greeter's SayHello service with something like:

object : GreeterGrpcKt.GreeterCoroutineImplBase() {
override suspend fun sayHello(request: HelloRequest) =
HelloReply
.newBuilder()
.setMessage("hello, ${request.name}")
.build()
}

You'll notice that the function has `suspend` on it because it uses Kotlin's coroutines, a built-in way to handle async / reactive IO. Check out the server example project.

With gRPC the client "stubs" are generated making it easy to connect to gRPC services. For the protoc above, the client stub can be used in Kotlin with:

val stub = GreeterCoroutineStub(channel)
val request = HelloRequest.newBuilder().setName("world").build()
val response = stub.sayHello(request)
println("Received: ${response.message}")

In this example the `sayHello` method is also a `suspend` function utilizing Kotlin coroutines to make the reactive IO easier. Check out the client example project.

Kotlin also has an API for doing reactive IO on streams (as opposed to requests), called Flow. gRPC Kotlin generates client and server stubs using the Flow API for stream inputs and outputs. The proto can define a service with unary streaming or bidirectional streaming, like:

service Greeter {
rpc SayHello (stream HelloRequest) returns (stream HelloReply) {}
}

In this example, the server's `sayHello` can be implemented with Flows:

object : GreeterGrpcKt.GreeterCoroutineImplBase() {
override fun sayHello(requests: Flow<HelloRequest>): Flow<HelloReply> {
return requests.map { request ->
println(request)
HelloReply.newBuilder().setMessage("hello, ${request.name}").build()
}
}
}

This example just transforms each `HelloRequest` item on the flow to an item in the output / `HelloReply` Flow.

The bidirectional stream client is similar to the coroutine one but instead it passes a Flow to the `sayHello` stub method and then operates on the returned Flow:

val stub = GreeterCoroutineStub(channel)
val helloFlow = flow {
while(true) {
delay(1000)
emit(HelloRequest.newBuilder().setName("world").build())
}
}

stub.sayHello(helloFlow).collect { helloResponse ->
println(helloResponse.message)
}

In this example the client sends a `HelloRequest` to the server via Flow, once per second. When the client gets items on the output Flow, it just prints them. Check out the bidi-streaming example project.

As you've seen, creating data transfer objects and services around them is made elegant and easy with gRPC Kotlin. But there are a few other exciting things we can do with this...

Android Clients

Protobuf compilers can have a "lite" mode which generates smaller, higher performance classes which are more suitable for Android. Since gRPC Kotlin uses gRPC Java it inherits the benefits of gRPC Java's lite mode. The generated code works great on Android and there is a `grpc-kotlin-stub-lite` artifact which depends on the associated `grpc-protobuf-lite`. Using the generated Kotlin stub client is just like on the JVM. Check out the stub-android example and android example.

GraalVM Native Image Clients

The gRPC lite mode is also a great fit for GraalVM Native Image which turns JVM-based applications into ahead-of-time compiled native images, i.e. they run without a JVM. These applications can be smaller, use less memory, and start much faster so they are a good fit for auto-scaling and Command Line Interface environments. Check out the native-client example project which produces a nice & small 14MB executable client app (no JVM needed) and starts, connects to the server, makes a request, handles the response, and exits in under 1/100th of a second using only 18MB of memory.

Google Cloud Ready

Backend services created with gRPC Kotlin can easily be packaged for deployment in Kubernetes, Cloud Run, or really anywhere you can run docker containers or JVM apps. Cloud Run is a cloud service that runs docker containers and scales automatically based on demand so you only pay when your service is handling requests. If you'd like to give a gRPC Kotlin service a try on Cloud Run:

  1. Deploy the app with a few clicks
  2. In Cloud Shell, run the client to connect to your app on the cloud:
    export PROJECT_ID=PUT_YOUR_PROJECT_ID_HERE
    docker run -it gcr.io/$PROJECT_ID/grpc-hello-world-mvn \
    "java -cp target/classes:target/dependency/* io.grpc.examples.helloworld.HelloWorldClientKt YOUR_CLOUD_RUN_DOMAIN_NAME"

Here is a video of what that looks like:

Check out more Cloud Run gRPC Kotlin examples

Thank You!

We are super excited to have reached 1.0 for gRPC Kotlin and are incredibly grateful to everyone who filed bugs, sent pull requests, and gave the pre-releases a try! There is still more to do, so if you want to help or follow along, check out the project on GitHub.

Also huge shoutouts to Brent Shaffer, Patrice Chalin, David Winer, Ray Tsang, Tyson Henning, and Kevin Bierhoff for all their contributions to this release!

Connecting and collaborating with Google Meet & G Suite

In this unprecedented time in history, we’ve seen millions of businesses across India adapt to the new normal of working remotely and learning from home. It is human nature to connect, and video conferencing plays a pivotal role here. 
We saw more than 3 million new users connecting on Google Meet every day as of this month, spending over 3 billion minutes a day together. That’s a massive 30x jump from the numbers we saw earlier in January. And today, we’re making our premium video conferencing tool, Google Meet free for everyone, with availability rolling out over the coming weeks. 
Stay home, stay safe, stay connected 
Starting May, anyone with an email address can create a Google Account and use Meet to schedule, join or start secure video meetings with anyone — whether it’s a virtual dance class, a weekly book club, neighborhood meetings, or any other reason to connect with your community, friends, and family. 
Until today, Google Meet was only available as part of G Suite, our collaboration and productivity solution for businesses, organizations and schools. Now, it is freely available on the web at meet.google.com and via mobile apps for iOS or Android. And if you use Google Calendar, you can easily start or join from there too. 
We’re also rolling out new features including tiled layout for larger calls, the option to present a Chrome tab (instead of just presenting their window or entire screen), low-light mode and eventually noise cancellation. 
Secure by design
Meet is designed, built and operated to be secure at scale — for everyone. We employ a vast array of safe-by-default measures to keep your meetings safe without doing a thing, everytime. We don't require or ask for any plugins to be installed, reducing the amount of software users and businesses need to patch with security updates on their machines. 
We also ensure that only authorised users can use and access Meet services by using a 2-Step Verification option for account — making them secure and convenient. Google Meet users can enroll their accounts in our Advanced Protection Program (APP), which provides our strongest protections available against phishing and account hijacking, and is specifically designed for the highest-risk accounts.
Helping businesses collaborate with G Suite
We’re not just connecting over video. We’ve also seen huge spikes in the use of our entire G Suite offering as more people create, share, and connect together while working remotely. Earlier this year we marked another major milestone — surpassing six million paying businesses and organisations who use G Suite. 
“Employees are able to access every business application via Google Cloud Platform and continue to communicate as usual not only between themselves but also with customers, vendors and other stakeholders with G Suite. The Google Cloud team is always accessible and supportive to help us ease things. The use of collaborative tools has facilitated important human contact and responsiveness in an unprecedented time of remote work,” said V M Samir, Group CIO, Rustomjee, a leading real estate company in Mumbai.
Mathan Babu Kasilingam, CISO of National Payments Corporation of India, an umbrella organisation for all retail payment systems in India says, “Google Meet has played a good role in helping our teams stay connected. It’s great to see that it is possible to work across various remote locations and manage to carry on business as usual through video conferencing.” 
Here is what TR Chadha & Co, one of India’s prominent chartered accountancy firms, had to say. “G Suite has been a lifeline for the teams for the past month since we have transitioned to a work from home set-up due to the pandemic. With G Suite, our teams can securely log-in from any device to work wherever they are at any time,” said Gautam Kumar, IT Manager, TR Chadha.
Securely stay connected and productive not just today but also in the future
We’re carefully rolling out Meet incrementally over the coming weeks to ensure we can provide everyone with the reliability and security they expect from Google. This means you might not be able to create meetings right away, but you can sign up to be notified when it’s available.
Meetings are limited to 60 minutes for the free product, though we will not enforce this time limit until after September 30.  Creating a trusted meeting space is important, so being mindful when sharing meeting links in public forums can help create a safe experience for all attendees. For more tips on how to use Meet securely and effectively, visit our Help Center.

Posted by Karan Bajwa, Managing Director, Google Cloud India

Growing our investment in India with a new cloud region coming to Delhi

At Google Cloud, our mission is to accelerate every organisation’s ability to transform through data-powered innovation with leading infrastructure, platform, industry solutions and expertise designed to meet our customers where they are on their journey to the cloud. 


Our cloud regions bring Google Cloud Platform (GCP) services to global organizations in industries like media and entertainment, retail and manufacturing, helping them drive growth, differentiation, and innovation for their business. As our customers in India grow and diversify, we continue to advance and invest in our cloud infrastructure to help regulated industries such as healthcare and financial services, as well as public sector organisations across India achieve their goals. 


Introducing our next cloud region in India


Today, we are thrilled to share our plans to open a Delhi cloud region. This will be our second cloud region in India since we launched our Mumbai region in 2017, and will expand our existing network of eight regions in Asia Pacific and 22 regions globally today.


GCP regions are the cornerstone of our cloud infrastructure, and they enable customers such as L&T Finance, Manipal Hospital Group, Reliance Mutual Fund, Royal Enfield, TechMahindra, Truecaller, and many more to deliver high performing, secure, low latency, cloud-based services to their users, no matter where they are around the world. 




Projected to launch in 2021, our Delhi cloud region will have three zones to protect against service disruptions. The region will also include  a portfolio of key GCP products, offer lower latency to nearby users, and, when combined with the existing region in Mumbai, enables geographically separate in-country disaster recovery for our customers’ mission critical applications. 


For customers in Delhi and beyond—and businesses and organisations yet to try our services—the new region presents an opportunity to take advantage of our big data and infrastructure services onshore while staying compliant with India’s data laws and regulations.  

Here’s what some of our customers have to say about the value of our expanding GCP infrastructure brings to their business:

“Cloud is the technology that enables us to achieve scale and reach. Today there are countless data points available about rural consumers which enable us to personalize our products to serve them better. With access to faster compute power, we can also on-board consumers more efficiently. Our rural businesses have clocked a disbursement CAGR of 60% over the past three years." 
-- Sunil Prabhune, Chief Executive-Rural Finance, and Group Head-Digital, IT and Analytics, L&T Financial Services
“We knew we made the right choice when we selected Dialogflow Enterprise Edition as our strategic, long-term platform to run intelligent chatbots across our inhouse processes and functions. A Delhi cloud region shows we are working with a cloud provider that shares our commitment to high quality in-market and multinational services.” -- Mukesh Rathi, CIO & Chief Digital Officer, Dr. Reddy's Laboratories
“Buyers and suppliers can already access our marketplace much faster than previously with Google Cloud, and this has a positive impact on customer engagement, time spent and the entire user journey. We are extremely excited about the potential of a second GCP region in India to help us provide an even better experience to the businesses that use IndiaMART.” -- Amarinder S Dhaliwal, Chief Product Officer, IndiaMART
“Google Cloud is transforming the cloud market similar to what NoBroker is doing by providing C2C platform for real estate needs without any brokerages. A second Google Cloud region in India will give us access to secure infrastructure closure to customers as we recently launched NoBroker services in NCR region. It will also help us retain customer data within India - meeting our compliance requirements.” -- Akhil Gupta, Co Founder and Chief Technology Officer, NoBroker

2020 is already shaping up to be a great year for Google Cloud in India, and we look forward to helping businesses and governments solve their most complex challenges and create value for years to come. 

Posted by Rick Harshman, Managing Director, Google Cloud Asia Pacific

W3C Trace Context Specification: What it Means for You

Since the first days of Google Cloud Platform (GCP), Google has been at the forefront of making your applications more observable. Beyond Stackdriver, our most visible impact in this space is OpenTelemetry, which we initiated in 2017 (as OpenCensus) and has grown into a huge community that includes the majority of APM / monitoring vendors and cloud platforms.

While OpenTelemetry allows developers to easily capture distributed traces and metrics from their own services, there’s also a need to trace requests as they propagate through components that developers don’t directly control, like managed services, load balancers, network hardware, etc. To solve this we co-defined a prototype HTTP header that these components can rely on, gathered partners, and moved the work into the W3C.

This work is now complete, and the W3C Trace Context format is now an official standard. Once implemented in GCP, this will make our services even easier to manage, both with Stackdriver and other third party distributed tracing tools. We explain more in the
official post on the W3C blog, which I’ve copied below:

The W3C Distributed Tracing working group has moved the Trace Context specification to the next maturity level. The specification is already being adopted and implemented by many platforms and SDKs. This article describes the Trace Context specification and how it improves troubleshooting and monitoring of modern distributed apps.

W3C Trace Context specification defines the format for propagating distributed tracing context between services. Distributed tracing makes it easy for developers to find the causes of issues in highly-distributed microservices applications by tracking how a single interaction was processed across multiple services. Each step of a trace is correlated through an ID that is passed between services, and W3C Trace Context now defines a standard for these context propagation headers.

Until now, different tracing systems have defined their own headers. Examples include Zipkin’s B3 format and X-Google-Cloud-Trace. Adopting a common context propagation format has been long desired by developers, APM vendors, and cloud platform hosts, as compatibility provides numerous benefits:
  • Web and RPC frameworks that use this standard to provide context propagation out of the box will also offer cross-service log correlation, even for developers who haven’t set up distributed tracing.
  • API producers can record the trace IDs of requests from API consumers and provide additional spans or metadata to their customers for a given traced request. Producers can also correlate customer trace IDs to internal traces when debugging technical issues raised by consumers.
  • Networking infrastructure (proxies, load balancers, routers, etc.) can both ensure that context propagation headers are not removed from requests passing through them, and can record spans or logs for a given trace, without having to support multiple vendor-specific formats. Potential examples of these include router appliances, cloud load balancers, and sidecar proxies like Envoy.
  • Instrumentation can be further decoupled from a developer’s choice of APM vendor. For example, using both OpenTelemetry and a given vendor’s agents, a developer can instrument different services in an application, and traces will flow through the system and be processed correctly by the vendor’s backend.
  • Web browsers and other clients can use these identifiers to correlate their telemetry with traces collected from backend services. This functionality is currently being defined.
To address this effort, a group of cloud providers, open source contributors, and APM vendors started defining a standard HTTP context propagation header that would replace their homegrown formats. This specification has been discussed and iterated on over the past two years, and the group working on it has grown significantly over that time. Sponsors include Google, Microsoft, Dynatrace, and New Relic (W3C members), and the group was officially moved into the W3C in 2018 for the work to proceed under the guidance of an official standards body and to spur even greater adoption.

TraceContext has since been adopted by OpenTelemetry (which enables it by default and also serves as the reference implementation), Azure services, Dynatrace, Elastic, Google Cloud Platform, Lightstep, and New Relic. We are tracking adoption in this list.

This first phase of work has focused on HTTP, as it is commonly used and has no built-in affordances for trace context propagation (gRPC and some newer RPC systems do). The same group of committee members are also working to define trace context propagation in other formats, starting with AMQP and MQTT for IoT; other upcoming topics include context propagation from clients and web browsers.

By Morgan McLean, OpenTelemetry + Stackdriver

An Inside Look at Google Earth Timelapse



Six years ago, we first introduced Google Earth Timelapse, a global, zoomable time-lapse video that lets anyone explore our changing planet’s surface—from the global scale to the local scale. Earth Timelapse consists of 83 million multi-resolution overlapping video tiles, which are made interactively explorable through the open-source Time Machine client software developed at Carnegie Mellon University’s CREATE Lab. At its core, Google Earth Timelapse is an example of how organizing information can make it more accessible and useful, turning petabytes of satellite imagery into an interactive experience that shows the dynamic changes occurring across space and time.
In April, we introduced several updates to Timelapse, including two additional years of imagery to the time-series visualization, which now spans from 1984 to 2018, with visual upgrades that make exploring more accessible and intuitive. We are especially excited that this update includes support for mobile and tablet devices, which are quickly overtaking desktop computers as the dominant source of app traffic.

Building the Global Visualization
Making a planetary-sized time-lapse video required a significant amount of pixel crunching in Earth Engine, Google's cloud platform for petabyte-scale geospatial analysis. The new release followed a process similar to what we did in 2013, but at a significantly greater scale—turning 15 million satellite images acquired over the last three and a half decades from the USGS/NASA Landsat and European Sentinel programs into 35 cloud-free 4-terapixel images of the planet—one for each year from 1984 to 2018.
At its native resolution, the Timelapse visualization is a 4 terapixel video (that's four trillion pixels), which would take about 12 days to download on a 95 Mb/s internet connection. Most computers would have difficulty playing a video of this size, let alone with an interactive, zoomable interface. The problem is even more severe for a mobile device.

A solution was pioneered by Google Maps in 2004 with the map pyramiding technique. Before that time, navigating a map required the use of directional arrows to pan and zoom, with each step requiring the page to reload. The map pyramiding technique assembles the full map image displayed on-screen from tens of small 256x256 pixel non-overlapping image tiles in an array, with new tiles fetched as needed at an appropriate resolution as the user pans and zooms across the map.
A traditional Mercator map pyramid contains non-overlapping image tiles.
This works very well for maps made of static images, but less so for pyramids of video tiles, such as those used by Timelapse, since it requires a web browser to keep up to 16 videos in sync while interacting with the visualization. The solution is embodied in CREATE Lab’s open source Time Machine software: create much larger video tiles that can cover the entire screen and only show one whole-screen tile at a time. The tiles create a pyramid, where sibling tiles overlap with their neighbors to provide a seamless transition between tiles while panning and zooming. Though the overlapping tiles require the use of about 16x more videos, this pyramid structure enables the use of Timelapse on mobile devices by minimizing the amount of data required for visualization.

In our newest release, the global video pyramid consists of 83 million videos across 13 zoom levels, which required about 2 million CPU hours distributed across thousands of machines in Google Cloud to generate.
Earth Timelapse uses a pyramid of overlapping video tiles.
Time Travel, Wherever You Are
Prior to April's update, ~30% of visitors to the Timelapse visualization were on mobile devices and didn't actually experience the visualization; instead they saw a YouTube playlist of locations in Timelapse. Until recently, the hardware and CPUs for phones and tablets could not decode videos fast enough without significant delays when someone attempted to zoom in or pan across a video, making mobile exploration unpleasant, if not impossible. In addition, in order for the visualization to be smooth as you pan and zoom, each video that is loaded must sync to the previously playing video and begin playing automatically. But, until only recently, mobile browser vendors had disabled video autoplay at the browser level for bandwidth reasons.

Now that mobile browser vendors have re-enabled video autoplay, we are able to take advantage of current mobile hardware and CPU capabilities, while leveraging the pyramid mapping technique’s efficient use of data, to enable Timelapse on mobile.

Redesigning Timelapse for Exploration Across Devices
Timelapse is a tool for exploration, so we designed for immersiveness, devoting as much real estate as possible to the map. On the other hand, it's not just a map, but a map of videos. So we kept controls visible, like pausing and restarting the timeline or choosing highlights, by leveraging Material Design with simple, clean lines and clear focal areas.
Navigate with Google Maps using the new "Maps Mode" toggle.
To explore, you need to know where you are or where somewhere else is, so the new interface includes a new "Maps Mode" toggle that lets the user navigate with Google Maps. We also built in scalability to the timeline element of the UI, so that new features added in the future, such as lengthening the time-lapse or adding options for different time increments, won’t break the design. The timeline also allows the user to go backwards in time—an interesting way to compare the present with the past.

For desktop browsers supporting WebGL, we also added a new WebGL viewer to the open source project, which loads and synchronizes multiple videos to fill the screen at optimal resolution. The aesthetic improvement of this is nontrivial, with >4x better resolution.

What's next
We're excited about the abundance of freely available, openly licensed satellite imagery and remote sensing data available, enabling new visualizations across time, space, and the visual and non-visual spectrum. We've found it's often the data combined with supplemental layers, such as the World Database on Protected Areas (WDPA) boundaries, that can spark new insights. For example, seeing the visual connection between declining home ownership and shifts in the city of Pittsburgh's racial makeup tells a story about inequality that numbers on a page simply cannot. Visual evidence can transcend language and cultural barriers and, we hope, generate productive conversations about our global challenges.

Acknowledgements
Randy Sargent, Senior Systems Scientist, Carnegie Mellon University CREATE Lab and the Google Earth Engine team

Source: Google AI Blog