Tag Archives: Firebase

Android Device Streaming, powered by Firebase, is now in Beta

Posted by Adarsh Fernando, Senior Product Manager, Android Developer Tools

Validating your app on a range of Android screens is an important step to developing a high quality Android app. However, getting access to the device you need, when you need it, can be challenging and time consuming. From trying to reproduce a device specific behavior on a Samsung device to testing your adaptive app layouts on the Google Pixel Fold, having the right device at the right time is critical.

To address this app developer use case, we created Android Device Streaming, powered by Firebase. With just a few clicks, you and your team can access real physical devices, such as the latest Pixel and Samsung devices, and use them in the IDE in many of the ways you would use a physical device sitting on your desk.

Animation of using Device Streaming in Android Studio
Android Device Streaming, powered by Firebase, available in Android Studio Jellyfish

Today, Android Device Streaming is in beta and is available to all Android developers using Android Studio Jellyfish or later. We’ve also added new devices to the catalog and introduced flexible pricing that provides low-cost access to the latest Android devices.

Read below to learn what changes are in this release, as well as common questions around uses, security, and pricing. However, if you want to get started right away and try Android Device Streaming at no cost, see our getting started guide.

What can you do with Android Device Streaming?

If you’ve ever used Device Mirroring, you know that Android Studio lets you see the screen of your local physical device within the IDE window. Without having to physically reach out to your device, you’re able to change the device orientation, change the posture of foldables, simulate pressing physical buttons, interact with your app, and more. Android Device Streaming leverages these same capabilities, allowing you to connect and interact with remote physical devices provided by Firebase.

Screen capture of using the debugger with Android Device Streaming
Using the Debugger with Android Device Streaming

When you use Android Studio to request a device from Android Device Streaming, the IDE establishes a secure ADB over SSL connection to the device. The connection also lets you use familiar tools in Android Studio that communicate with the device, such as the Debugger, Profiler, Device Explorer, Logcat, Compose Live Edit, and more. These tools let you more accurately validate, test, and debug the behavior of your app on real OEM hardware.

What devices would my team have access to?

Android Device Streaming gives you and your team access to a number of devices running Android versions 8.1 through 14. You have access to the latest flagship devices from top device manufacturers, such as Google Pixel and Samsung. You can expand testing your app across more form factors with access to the latest foldables and tablets, such as the Samsung Tab S8 Ultra.

Screen capture of browsing the list of devices and selecting the one you want to use in Android Studio
Browse and select devices you want to use from Android Studio

And we’re frequently adding new devices to our existing catalog of 20+ device models, such as the following recent additions:

    • Samsung Galaxy Z Fold5
    • Samsung Galaxy S23 Ultra
    • Google Pixel 8a

Without having to purchase expensive devices, each team member can access Firebase’s catalog of devices in just a few clicks, for as long as they need—giving your team confidence that your app looks great across a variety of popular devices.


Google OEM partner logos - Samsung, Google Pixel, Oppo, and Xiaomi

As we mentioned at Google I/O ‘24, we’re partnering with top Original Equipment Manufacturers (OEMs), such as Samsung, Google Pixel, Oppo, and Xiaomi, to expand device selection and availability even further in the months to come. This helps the catalog of devices grow and stay ahead of ecosystem trends, so that you can validate that your apps work great on the latest devices before they reach the majority of your users.

Is Android Device Streaming secure?

Android Device Streaming, powered by Firebase, takes the security and privacy of your device sessions very seriously. Firebase devices are hosted in secure global data centers and Android Studio uses an SSL connection to connect to the device.

A device that you’ve used to install and test your app on is never shared with another user or Google service before being completely erased and factory reset. When you’re done using a device, you can do this yourself by clicking “Return and Erase Device” to fully erase and factory reset it. The same applies if the session expires and the device is returned automatically.

Screen capture of Reuturn and Erase Device function in Android Device Streaming
When your session ends, the device is fully erased and factory reset.

How much does Android Device Streaming cost?

Depending on your Firebase project’s pricing plan, Android Device Streaming users can use Android Device Streaming with the following pricing:

    • On June 1, 2024, for a promotional period:
        • (no cost) Spark plan: 120 no cost minutes per project, per month
        • Blaze plan: 120 no cost minutes per project, per month, 15 cents for each additional minute
    • On or around February, 2025, the promotional period will end and billing will be based on the following quota limits:
        • (no cost) Spark plan: 30 no cost minutes per project, per month
        • Blaze plan: 30 no cost minutes per project, per month, 15 cents for each additional minute

With no monthly or yearly contracts, Android Device Streaming’s per-minute billing provides unparalleled flexibility for you and your team. And importantly, you don’t pay for any period of time required to set up the device before you connect, or erase the device after you end your session. This allows you and your team to save time and costs compared to purchasing and managing your own device lab.

To learn more, see Usage levels, quotas, and pricing.

What’s next

We’re really excited for you and your team to try Android Device Streaming, powered by Firebase. We think it’s an easy and cost-effective way for you to access the devices you need, when you need them, and right from your IDE, so that you can ensure the best quality and functionality of your app for your users.

The best part is, you can try out this new service in just a few clicks and at no cost. And our economical per-minute pricing provides increased flexibility for your team to go beyond the monthly quota, so that you only pay for the time you’re actively connected to a device—no subscriptions or long-term commitments required.

You can expect that the service will be adding more devices from top OEM partners to the catalog, to ensure that device selection remains up-to-date and becomes increasingly diverse. Try Android Device Streaming today and share your experience with the Android developer committee on LinkedIn, Medium, YouTube, or X.

Google I/O 2024: What’s new in Android Development Tools

Posted by Mayank Jain – Product Manager, Android Studio

At Google I/O 2024, we announced an exciting new set of features and tools aimed at making Android development faster and easier. We also shared updates to Android Studio that will help you leverage AI and make it easier for you to build high quality apps for Android across the Android ecosystem.

You can check out the What’s new in Android Developer Tools session at Google I/O 2024 to see some of the new features in action or better yet, try them out yourself by downloading Android Studio Koala 🐨 Feature Drop in the preview release channel. Here’s a look at our announcements:

Leverage Gemini in Android Studio

Since launching AI features in Android Studio last year, we continue to evolve our underlying models, integrate your feedback, and expand availability to more countries and territories so that you can leverage AI in your workflow and become a more productive Android app developer. Using the built-in AI privacy controls, you can opt in to using the latest AI feature improvements that are tailored for your Android app project.

Code suggestions with Gemini in Android Studio

You can now provide custom prompts for Gemini in Android Studio to generate code suggestions. After you enable Gemini from the View > Tool Windows > Gemini tool window, right-click in the code editor and select Gemini > Transform selected code from the context menu to see the prompt field. You can then prompt Gemini to generate a code suggestion that either adds new code or transforms selected code. You can ask Gemini to simplify complex code by rewriting it, perform very specific code transformations such as “make this code idiomatic,” or generate new functions you describe. Android Studio then shows you Gemini’s code suggestion as a code diff, so that you can review and accept only the suggestions you want.

Code suggestions with Gemini in Android Studio

Gemini for recommendations on crash reports

App Quality Insights in Android Studio seamlessly incorporates both Firebase Crashlytics and Android Vitals data into Android Studio so you can access the most important app stability related information, without having to switch tools.

You can now use Gemini in Android Studio to analyze your crash reports, generate insights which are shown in the Gemini tool window, provide a crash summary, and when possible recommend next steps, including sample code and links to relevant documentation.

You can generate all of this information directly from the App Quality Insights tool window in Android Studio after you enable Gemini from View > Tool Windows > Gemini.

Gemini for recommendations on crash reports

Integrate Gemini API into your app with a starter template

Start prototyping with Gemini models in your apps with our new starter app template provided in Android Studio. In this app template, you can issue prompts directly to the Gemini API, add image sources as input, and display the responses on the screen. Additionally, use Google AI Studio to craft custom prompts for your app.

When you are ready to scale your AI features to production with Google Cloud infrastructure, you can also access the powerful capabilities of Gemini models through Vertex AI. This is Google’s fully-managed development platform designed for building and deploying generative AI. Whether you simply need world class inference capabilities, or want to build end-to-end AI workflows with Vertex, the Gemini API is a great solution.

Integrate Gemini API into your app with a starter template

Gemini 1.5 Pro coming to Android Studio

We previously announced that Gemini in Android Studio uses the Gemini 1.0 Pro model to help you by answering Android development questions, generating code, finding resources, or explaining best practices. In this preview stage of Gemini in Android Studio, we are offering Gemini 1.0 Pro at no-cost for all users for now. Gemini 1.0 Pro is a versatile model, making it ideal to scale. However we acknowledge that its quality of responses may be limited in some cases. Based on your feedback, we are committed to improving the quality for Android development, and excited to add more features using Gemini to make your developer experience even more productive.

Along this journey, the Gemini 1.5 Pro model will be coming to Android Studio later this year. Equipped with a Large Context Window, this model notably leads to higher quality responses, and unlocks use cases like multimodal input that you might have seen in the Google I/O 2024 sessions. Stay tuned for more updates on how you can access more capable models in Android Studio.

Productivity enhancements

Release Monitoring with Firebase

Today we announced the general availability of the Firebase Release Monitoring Dashboard. The Firebase Release Monitoring Dashboard is a single dashboard powered by Firebase Crashlytics to monitor your most recent production releases of your Android app. It updates in real time to give you a high-level view of the most important release metrics, like crash-free sessions, comparisons, and benchmarking based on your previous releases.

Android Device Streaming

Android Device Streaming, powered by Firebase, lets you securely connect to remote physical Android devices hosted in Google's data centers. It is a convenient way to test your app against physical units of some of the latest Android devices, including the Google Pixel 8 and 8 Pro, Pixel Fold, and more.

Starting today, Android Device Streaming now includes the following devices, in addition to the portfolio of 20+ device models already available:

    • Samsung Galaxy Fold5
    • Samsung Galaxy S23 Ultra
    • Google Pixel 8a

Additionally, if you’re new to Firebase, Android Studio automatically creates and sets up a no-cost Firebase project for you when you sign in to Koala Feature Drop to use Device Streaming. So, you can get to streaming the device you need much faster. Learn more about Android Device Streaming quotas, including promotional quota for the Firebase Blaze plan projects available for a limited time.

Connect to the latest physical Android devices in moments with Android Device Streaming, 
powered by Firebase

USB cable speed detection

Did you know that USB cable bandwidth varies from 480 Mbps (USB-2) to up to 40,000 Mbps (USB-4)? Android Studio Koala Feature Drop now makes it trivial to differentiate low performing USB cables from the high performing ones.

When you connect an Android device, Android Studio automatically detects the device and USB cable bandwidth and warns you if there’s a mismatch in USB bandwidth.

Note: USB cable speed detection requires an updated ADB found in Android SDK Platform Tools v34+, and is currently available for macOS and Linux.

USB cable speed detection.
Learn more about USB speeds here

A new way to sign in with Google in Android Studio

It’s now easier to sign in to multiple Google services with one authentication step. Whether you want to use Gemini in Android Studio, Firebase for Android Device Streaming, Google Play for Android Vitals reports, or all these useful services, the new sign in flow makes it easier to get up and running. If you’re new to Firebase and want to use Android Device Streaming, Android Studio automatically creates a project for you, so you can quickly start streaming a real physical Firebase device. With granular permissions scoping, you will always be in control of which services have access to your account. To get started, just click the profile avatar and sign in with your developer account.

A new way to sign in with Google in Android Studio

Device UI setting shortcut

Using the device UI setting shortcut, you can now effortlessly configure your devices to desired settings related to dark theme, font size, display size, app language, and more, all directly through the Running Devices window. You can now test and debug your UI seamlessly for any of the possible scenarios required by your use case.

Device UI settings shortcuts

Faster and improved Profiler with a task-centric approach

The internals of the Android Studio Profiler have been dramatically improved. Popular profiling tasks like capturing a system trace with profileable apps now start up to 60% faster.*

We’ve redesigned the profiler to make it easier to start the task you’re interested in, whether it’s profiling your app’s CPU, memory, or power usage. For example, initiating a system trace task to profile and improve your app’s startup time is integrated right in the UI as you open the profiler.

Faster and improved Profiler with a task-centric approach 
*Based on internal data, as tested in April 2024

Google Play SDK Index integration

Android Studio is integrated with the Google Play SDK Index to inform when there are known policy or version issues with SDKs used by your app. This enables you to update those dependencies and avoid issues that could prevent you from publishing new versions of your app.

In the Android Studio Koala Feature Drop release, the integration has been expanded to also include warnings from the Google Play SDK Console. This gives you a complete view of any potential version or policy issues in your dependencies before submitting your app to the Google Play Console.

Notes from SDK authors are now also displayed directly in Android Studio to save you time.

A warning from the SDK Index with the corresponding SDK author note

Preview tiles for Wear OS apps

Android Studio now has preview support for Tiles. You can now iterate much quicker when creating tiles, enabling you to quickly see what a Tile looks like on different configurations without needing to run it on a device.

Tiles previews usage for Wear OS apps

Generate synthetic sensor data for testing on Wear OS apps

To help simulate real life scenarios you can now generate synthetic (fake) data for a Wear OS emulator for health related sensors such as heart rate, speed, steps, and more. You are now able to set up and perform testing for a multi-sport training session in minutes, end-to-end in Android Studio, without ever leaving your desk.

Generate synthetic sensor data for testing on Wear OS apps

Compose Glance widget previews

Android Studio Koala Feature Drop makes it easy to preview your Jetpack Compose Glance widgets (1.1.0-rc01) directly within the IDE. Catch potential UI issues and fine-tune your widget's appearance early in the development process. Learn more about how to get started.

Previews for Compose Glance widgets

Live Edit for Compose enabled by default

Live Edit for Compose can accelerate your Compose development experience by automatically deploying code changes to the running application on an emulator or physical device. Live Edit can help you see the effect of updates to UX elements—for example new composables, modifier updates, and animations—on the overall app experience. As you become more familiar with Live Edit you will find many creative ways it can help improve your development experience and productivity.

In Android Studio Koala Feature Drop, Live Edit is enabled by default in manual mode and has increased stability and more robust change detection, including support for import statements.

ALT TEXT
Compose Preview Screenshot Testing with Now in Android app

Compose preview screenshot testing plugin (alpha)

Host-side screenshot testing is an easy and powerful way to test UIs and prevent regressions. Today, the first alpha version of the Compose Preview Screenshot Testing plugin is available as a separate plugin, to be used together with AGP 8.5.0-beta01 or higher. Add your Compose Previews to the src/main/screenshotTest folder and run the task to generate a diff report after UI updates. The generated HTML test report lets you visually detect any changes to your app’s UI.

This alpha version of the plugin is designed for rapid iteration and feedback. We plan to merge it back into AGP in the future, but for now, this separate plugin lets us experiment and improve the feature quickly. Learn more about how to get started.

IntelliJ Platform Update (2024.1)

Android Studio Koala Feature Drop includes the IntelliJ 2024.1 platform release, which comes with some very useful IDE improvements:

    • An overhauled terminal featuring both visual and functional enhancements to streamline command-line tasks. Learn more in this blog post.
    • A new feature called sticky lines in the editor simplifies working with large files and exploring new codebases. This feature keeps key structural elements, like the beginnings of classes or methods, pinned to the top of the editor as you scroll and provides an option to promptly navigate through the code by clicking on a pinned line.
    • Basic IDE functionalities like code highlighting and completion now work for Java and Kotlin during project indexing, which should enhance your startup experience.
    • You can now scale the IDE down to 90%, 80%, or 70%, giving you the flexibility to adjust the size of IDE elements both upward and downward.

Read the detailed IntelliJ release notes here.

To summarize

Android Studio Koala Feature Drop (2024.1.2) is now available in the Android Studio canary channel with

    • Gemini in Android Studio
        • Code suggestions with Gemini in Android Studio
        • Gemini for recommendations on crash reports
        • Gemini API starter app template to help integrate Gemini into your app (also available in Koala 2024.1.1)

    • Productivity enhancements
        • Release Monitoring with Firebase
        • Android Device Streaming
        • USB cable speed detection
        • A new way to sign in with Google in Android Studio
        • Device UI setting shortcut
        • Faster and improved Profiler with a task-centric approach
        • Google Play SDK Index integration
        • Preview tiles for Wear OS apps
        • Generate synthetic sensor data for testing on Wear OS apps
        • Compose Glance widget previews
        • Live Edit for Compose enabled by default
        • Compose preview screenshot testing plugin (alpha) - to be installed additionally

    • IntelliJ Platform Update (2024.1): also available in Koala 2024.1.1
        • An overhauled terminal
        • Sticky lines in editor simplifies working with large files
        • Code highlighting and completion now work during project indexing
        • Flexible IDE size adjustments

And last, a quick reminder that going forward, the initial Android Studio releases will have the .1 Android Studio major version and introduce the updated IntelliJ platform version, while subsequent Feature Drops will increase the Android major version to .2 and focus on introducing Android-specific features that help you be more productive for Android app development.

How to get started

Ready to try the exciting new features in Android Studio?

You can download the canary version Android Studio Koala 🐨 Feature Drop (2024.1.2) today to incorporate these new features into your workflow or try the stable version Android Studio Jellyfish 🪼. You can also install them side by side by following these instructions.

As always, your feedback is important to us – check known issues, report bugs, suggest improvements, and be part of our vibrant community on LinkedIn Medium, YouTube, or X. Let's build the future of Android apps together!

Global developers use Google tools to build solutions in recruiting, mentorship and more

Posted by Lyanne Alfaro, DevRel Program Manager, Google Developer Studio

Developer Journey is a monthly series highlighting diverse and global developers sharing relatable challenges, opportunities, and wins in their journey. Every month, we will spotlight developers around the world, the Google tools they leverage, and the kinds of products they are building.

This month we speak with global developers across Google Developer Experts, and Women Techmakers, to learn more about their favorite Google tools, the applications they’ve built to serve diverse communities and the role of inclusive design in their process.


Miguel Ángel Durán Garcí

Headshot of Miguel Ángel Durán Garcí, smiling
Barcelona, Spain
Google Developer Expert, Web Technologies
Content Creator & Software Engineer

What Google tools have you used to build?

I've been using Firebase, Google Cloud Platform, CrUX Dashboard, and Chrome DevTools for years. As a web developer, I'm always excited about the new features that DevTools brings to us to improve our productivity and the performance of our applications.


Which tool has been your favorite to use? Why?

Lately, I've been trying Project IDX, an entirely web-based workspace for full-stack application development, and I'm really excited about the future of this project. I love the idea of being able to develop and deploy applications from the browser, without having to install anything on my computer.


Please share with us about something you’ve built in the past using Google tools.

Most recently, I've deployed AdventJS, a holiday calendar for developers. For optimizing the images, I've used Squoosh from the GoogleChromeLabs team. To ensure the website was accessible and to tweak performance, I've used Lighthouse from Chrome DevTools. Also, I used Google Bard to translate the content of the website into English and Portuguese.


What will you create with Google Bard?

I'm planning to expand a website I've created for the Spanish-speaking community to teach JavaScript from scratch. With Google Bard, I can check the content, create some code, and make it help me create challenges for the students.


What advice would you give someone starting in their developer journey?

I would tell them to be patient and to enjoy the process. It's a long journey, but it's worth it. Also, I would tell them to be curious and avoid sticking to only a few technologies. And finally, I would tell them to share their knowledge with the community, because it's the best way to learn and meet new people. You don't need to be an expert to share your knowledge; you just need to be one step ahead of the people you're teaching.


Marian Villa

Headshot of Marian Villa, smiling
Medellín, Colombia
Google Developer Expert, Web Technologies
Co-founder / Director Pionerasdev

What Google tools have you used to build?

Development and Creativity:

  • Google Chrome DevTools
  • Bard
  • TensorflowJS

Productivity and Communication:

  • Gmail
  • Google Calendar
  • Google Drive
  • Google Docs
  • Google Sheets
  • Google Slides
  • Google Meet

Marketing and Business:

  • Google Ads
  • Google Analytics
  • Google My Business
  • Google Workspace
  • Google Cloud Platform
  • Google Marketing Platform

Education and Learning:

  • Google Classroom
  • Google Forms
  • Google Sites
  • YouTube

Which tool has been your favorite to use? Why?

Choosing a favorite tool is quite a task given the unique strengths of Bard, TensorflowJS and Google Chrome DevTools, but I'd have to say that Google Chrome DevTools stands out for me. Its versatility in inspecting and debugging web pages, testing code variations, and providing insights into JavaScript behavior has been crucial in my web development endeavors. That being said, both Bard and TensorFlow.js have incredible capabilities. Bard plays a vital role in generating creative content, answering queries, and even composing code. TensorFlow.js, on the other hand, is a game-changer, enabling machine learning in JavaScript, and making it accessible for a wide range of applications. Each tool has its unique appeal, and the choice will depend on the context and specific requirements of the task at hand.


Please share with us about something you’ve built in the past using Google tools.

On our latest website, we use all the Google technologies at hand to enhance our image as an NGO. Find it here.


What will you create with Google Bard?

We are once again resuming a winning mentorship project to advance our career as developers, so Bard and Duet AI are great allies to inspect our code and once again create an MVP of this product for our community.


What advice would you give someone starting in their developer journey?

First, think about the problem you want to solve, or what you want to contribute to the world, then create and make it come true. This is easier if you rely on communities, and people who help you as mentors, sponsors and guides.


Rubens de Almeida Zimbres

Headshot of Rubens Zimbres, smiling
São Paulo - Brazil
Google Developer Expert, Machine Learning and Google Cloud
ML Engineer

What Google tools have you used to build?

I’ve been using the full stack of Google Products. I use Google Workspace daily in my life, my personal website is made on Google Sites, and Google Cloud; I started with Compute Engine and Jupyter Notebooks, customized to my needs.

As I acquired more knowledge through practical experience, Coursera and Google Cloud Skills Boost, I started building end to-end solutions using BigQuery, SQL, lots of Vertex AI (Generative AI Studio, Matching Engine, Speech-to-text, Pipelines, AutoML, Model Fine-Tuning), Cloud Run (and a little GKE - Kubernetes), Cloud Functions, Dialogflow and Document AI.

As the requirements of clients change according to the industry, like recruiting (Virtual Career Center) and contact center (Contact Center AI), I was able to test and deploy in production different Google products to solve the clients’ needs.


Which tool has been your favorite to use? Why?

Vertex AI is my favorite, as it is pure ML and Deep Learning optimized. Using AutoML with NAS (Neural Architecture Search) was a very interesting experience with awesome results. Developing Machine Learning pipelines with Kubeflow is a special pleasure, as this is going into production and the whole MLOps is involved.


Please share with us about something you’ve built in the past using Google tools.

I’ve built a recruiting solution that was implemented in six countries of Latin America, benefiting more than 365,000 people. This solution automatically analyzes resumes using OCR via Document AI.

I delivered a revenue prediction for a hotel chain using Tensorflow, where we increased the accuracy of the client’s model by 0.95%. I also built a Contact Center solution which uses Google Speech-to-Text and analytics to make management easier and also to generate strategic insights.

Lately, I was part of the team that delivered an end-to-end Virtual Career Center solution that matches job candidates to job vacancies using Vertex AI Matching Engine via text embeddings and SCANN. Both the recruiting solution and the contact center solution generated patents in Brazil, in the field of NLP (Natural Language Processing).


What will you create with Google Bard?

Google Bard is part of my daily routine. It helps me while coding, it helps me to plan trips, get to the right public transportation, visit interesting places around the world and it also helps by retrieving the Google search in an organized way, with updated content. My idea is to use Bard along with LangChain to perform optimizations in the finance industry.


What advice would you give someone starting in their developer journey?

Learn the basics first.

The temptation of learning this magnificent field as Machine Learning is gigantic, but coding is a great part of the solution. Learn to code properly, in whatever language you want. This brings efficiency and security if your solution needs to scale, decreasing infrastructure costs and improving user experience.

The same applies to Machine Learning: learn basic disciplines such as Calculus, Computer Science fundamentals and you will understand most of the content is shared today online. Only after learning ML you should dive into Deep Learning and the disciplines associated. Don’t fake it. Make it.

Global Google Developer Experts Share Their Favorite Tools and Advice for New Developers

Posted by Lyanne Alfaro, DevRel Program Manager, Google Developer Studio

Developer Journey is a monthly series highlighting diverse and global developers sharing relatable challenges, opportunities, and wins in their journey. Every month, we will spotlight developers around the world, the Google tools they leverage, and the kinds of products they are building.

This month we speak with global Google Developer Experts in Firebase, Women Techmakers, and beyond, to learn more about their favorite Google tools, the applications they’ve built to serve diverse communities, and their best advice for anyone just getting started as a developer.

Juan Lombana

Headshot of Juan Lombana, smiling
Mexico City, Mexico
Founder, Mercatitlán

What Google tools have you used to build?

Google Analytics and Firebase's A/B testing features have been pivotal in our data-driven approach, enabling continuous improvement in our conversion strategies. More recently, Bard has become a significant asset in developing new products and in our educational endeavors, especially with the introduction of our AI course. Its utility in both product development and educational settings is profound.


Which tool has been your favorite to use? Why?

If I had to choose, it would be Google Ads. Its ability to consistently drive new customers and provide unparalleled visibility to quality products is unmatched. While it may not traditionally be considered a 'tool' in the strictest sense, its impact on business growth and visibility is indisputable.


Please share with us about something you’ve built in the past using Google tools.

My entire business, Mercatitlán, has been built and scaled using Google Tools. We have cultivated a community of over 40,000 paid students, educating them on effective use of Google Ads, leveraging Bard for enhanced website content, and employing Google Analytics for strategic A/B testing to boost sales. The transformational impact of these tools on both my business and my students' ventures is a testament to their potential.


What will you create with Google Bard?

The integration of Bard AI into our daily operations is revolutionizing the way we approach digital marketing. Beyond its current uses in social media content creation, ad ideas generation, email composition, and customer support enhancement, we're exploring several innovative applications:

  • Personalized Marketing Campaigns: Using Bard AI, we can analyze customer data and preferences to create highly personalized marketing campaigns. This helps in delivering more relevant content to our audience, thereby increasing engagement and conversion rates. 
  • Competitive Analysis: By analyzing competitor data, Bard AI can help us understand their strategies, strengths, and weaknesses. This intelligence is crucial for refining our marketing approach and differentiating our brand in the marketplace.
  • Content Optimization for SEO: Bard can assist in optimizing website and blog content for search engines. By understanding and integrating key SEO principles, it can help us rank higher in search results, thus improving our online visibility. 
  • Automated Reporting and Insights: Automating the generation of marketing reports and insights with Bard saves time and resources, allowing our team to focus on strategy and creativity rather than manual data analysis.

What advice would you give someone starting in their developer journey?

The key is to start with action rather than waiting for perfection. Adopt a mindset focused on experimentation and analytics. This approach allows you to follow data-driven insights rather than solely relying on innovation, leading to significant societal impact through technology.


Jirawat Karanwittayakarn

Headshot of Jirawat Karanwittayakarn, smiling
Bangkok, Thailand
Tech Evangelist, LINE Thailand

What Google tools have you used to build?

I have used a variety of Firebase services to build LINE chatbots for a number of years. These services have included Cloud Functions, Cloud Firestore, Cloud Storage, Firebase Hosting, and etc. Recently I have also used the PaLM API, a very powerful tool that allows me to build Generative AI chatbots.


Which tool has been your favorite to use? Why?

Firebase is my favorite tool because it is a platform that provides a complete set of tools for building and managing mobile, web, and chatbots. It is very easy to use and has a wide range of features that make it a great choice for developers of all levels. Furthermore, Firebase services have allowed me to scale my chatbots and make them more reliable.


Please share with us about something you’ve built in the past using Google tools.

  • LINE Developers TH is a chatbot that allows Thai developers to learn about LINE APIs and get started with building services. It also provides users with the ability to try out demos of LINE APIs.
  • TrueMoney is a wallet app that I have built in the past using Firebase. The app allows users to store money, send money, and pay bills. It is a very popular app in Thailand, with over 10 million users.
  • Sanook is an app that allows users to access news, articles, and other content from the number one web portal in Thailand on their mobile devices.

What will you create with Google Bard?

I would like to create a use case of building a powerful LINE chatbot using PaLM API and Firebase for developers. I believe this will be a great way to showcase the power of these tools and how they can be used to create innovative solutions.


What advice would you give someone starting in their developer journey?

First and foremost, I would encourage them to be curious and always be willing to learn new things. The world of technology is constantly changing, so it's important to stay up-to-date on the latest trends and technologies. This can be done by reading articles, attending conferences, and taking online courses.

Secondly, I would recommend that they find a mentor or role model who can help guide them on their journey. Having someone who has been through the process can be invaluable in providing support and advice. They can help you identify areas where you need to improve, and provide you with tips and tricks for success.

Finally, I would encourage them to never give up. The road to becoming a developer can be challenging, but it's also incredibly rewarding. If you're passionate about technology, then don't let anything stop you from pursuing your dreams.


Laura Morinigo

Headshot of Lauren Moringo, smiling
London, England
Women Techmakers Ambassador
Principal Engineer and Consultant, Samsung Electronics UK

What Google tools have you used to build?

I have used tools like Google Cloud and Firebase.


Which tool has been your favorite to use? Why?

I would say Firebase! It helped me to build web apps and explore new technologies easily while saving a lot of time and resources. Additionally, a lot of functionalities have been added recently. Over the years, I've witnessed its evolution, with the addition of numerous functionalities that continually enhance its utility and user experience. This constant innovation within Firebase not only simplifies complex tasks but also opens doors to creative possibilities in web app development.


Please share with us about something you’ve built in the past using Google tools.

I've been leading a project in partnership with the United Nations to help share information about its worldwide global goals. We used Firebase hosting and Cloud functions for the first release of the web app and it was a success! It felt very good to help create tools that support a good cause.


What will you create with Google Bard?

I'm experimenting with the current extensions to improve personal productivity. It's very interesting how you can improve the way that you do your daily tasks.


What advice would you give someone starting in their developer journey?

Remember that as a developer you will have the power to create! Use this power to build personal projects and combine it with things that you enjoy. You will start building a portfolio and have fun while learning. Finally, don't hesitate to find a mentor and connect with a community of developers to support and guidance in your journey. You can find a lot of help, improve your networking, and even have friends for life!

Full-stack development in Project IDX

Posted by Kaushik Sathupadi, Prakhar Srivastav, and Kristin Bi – Software Engineers; Alex Geboff – Technical Writer

We launched Project IDX, our experimental, new browser-based development experience, to simplify the chaos of building full-stack apps and streamline the development process from (back)end to (front)end.

In our experience, most web applications are built with at-least two different layers: a frontend (UI) layer and a backend layer. When you think about the kind of app you’d build in a browser-based developer workspace, you might not immediately jump to full-stack apps with robust, fully functional backends. Developing a backend in a web-based environment can get clunky and costly very quickly. Between different authentication setups for development and production environments, secure communication between backend and frontend, and the complexity of setting up a fully self-contained (hermetic) testing environment, costs and inconveniences can add up.

We know a lot of you are excited to try IDX yourselves, but in the meantime, we wanted to share this post about full-stack development in Project IDX. We’ll untangle some of the complex situations you might hit as a developer building both your frontend and backend layers in a web-based workspace — developer authentication, frontend-backend communication, and hermetic testing — and how we’ve tried to make it all just a little bit easier. And of course we want to hear from you about what else we should build that would make full-stack development easier for you!


Streamlined app previews

First and foremost, we've streamlined the process of enabling your applications frontend communication with its backend services in the VM, making it effortless to preview your full-stack application in the browser.

IDX workspaces are built on Google Cloud Workstations and securely access connected services through Service Accounts. Each workspace’s unique service account supports seamless, authenticated preview environments for your applications frontend. So, when you use Project IDX, application previews are built directly into your workspace, and you don’t actually have to set up a different authentication path to preview your UI. Currently, IDX only supports web previews, but Android and iOS application previews are coming soon to IDX workspaces near you.

Additionally, if your setup necessitates communication with the backend API under development in IDX from outside the browser preview, we've established a few mechanisms to temporarily provide access to the ports hosting these API backends.


Simple front-to-backend communication

If you’re using a framework that serves both the backend and frontend layers from the same port, you can pass the $PORT flag to use a custom PORT environment variable in your workspace configuration file (powered by Nix and stored directly in your workspace). This is part of the basic setup flow in Project IDX, so you don’t have to do anything particularly special (outside of setting the variable in your config file). Here’s an example Nix-based configuration file:


{ pkgs, ... }: {

# NOTE: This is an excerpt of a complete Nix configuration example.

# Enable previews and customize configuration
idx.previews = {
  enable = true;
  previews = [
    {
      command = [
        "npm"
        "run"
        "start"
        "--"
        "--port"
        "$PORT"
        "--host"
        "0.0.0.0"
        "--disable-host-check"
      ];
      manager = "web";
      id = "web";
    }
  ];
};

However, if your backend server is running on a different port from your UI server, you’ll need to implement a different strategy. One method is to have the frontend proxy the backend, as you would with Vite's custom server options.

Another way to establish communication between ports is to set up your code so the javascript running on your UI can communicate with the backend server using AJAX requests.

Let’s start with some sample code that includes both a backend and a frontend. Here’s a backend server written in Express.js:


import express from "express";
import cors from "cors";


const app= express();
app.use(cors());

app.get("/", (req, res) => {
    res.send("Hello World");
});

app.listen(6000, () => {
    console.log("Server is running on port 6000");
})

The bolded line in the sample — app.use(cors()); — sets up the CORS headers. Setup might be different based on the language/framework of your choice, but your backend needs to return these headers whether you’re developing locally or on IDX.

When you run the server in the IDX terminal, the backend ports show up in the IDX panel. And every port that your server runs on is automatically mapped to a URL you can call.

Moving text showing the IDX terminal and panel

Now, let's write some client code to make an AJAX call to this server.


// This URL is copied from the side panel showing the backend ports view
const WORKSPACE_URL = "https://6000-monospace-ksat-web-prod-79679-1677177068249.cluster-lknrrkkitbcdsvoir6wqg4mwt6.cloudworkstations.dev/";

async function get(url) {
  const response = await fetch(url, {
    credentials: 'include',
  });
  console.log(response.text());
}

// Call the backend
get(WORKSPACE_URL);

We’ve also made sure that the fetch() call includes credentials. IDX URLs are authenticated, so we need to include credentials. This way, the AJAX call includes the cookies to authenticate against our servers.

If you’re using XMLHttpRequest instead of fetch, you can set the “withCredentials” property, like this:


const xhr = new XMLHttpRequest();
xhr.open("GET", WORKSPACE_URL, true);
xhr.withCredentials = true;
xhr.send(null);

Your code might differ from our samples based on the client library you use to make the AJAX calls. If it does, check the documentation for your specific client library on how to make a credentialed request. Just be sure to make a credentialed request.


Server-side testing without a login

In some cases you might want to access your application on Project IDX without logging into your Google account — or from an environment where you can’t log into your Google account. For example, if you want to access an API you're developing in IDX using either Postman or cURL from your personal laptops's command line. You can do this by using a temporary access token generated by Project IDX.

Once you have a server running in Project IDX, you can bring up the command menu to generate an access token. This access token is a short-lived token that temporarily allows you to access your workstation.

It’s extremely important to note that this access token provides access to your entire IDX workspace, including but not limited to your application in preview, so you shouldn’t share it with just anyone. We recommend that you only use it for testing.

Generate access token in Project IDX

When you run this command from IDX, your access token shows up in a dialog window. Copy the access token and use it to make a cURL request to a service running on your workstation, like this one:


$ export ACCESS_TOKEN=myaccesstoken
$ curl -H "Authorization: Bearer $ACCESS_TOKEN" https://6000-monospace-ksat-web-prod-79679-1677177068249.cluster-lknrrkkitbcdsvoir6wqg4mwt6.cloudworkstations.dev/
Hello world

And now you can run tests from an authenticated server environment!


Web-based, fully hermetic testing

As we’ve highlighted, you can test your application’s frontend and backend in a fully self-contained, authenticated, secure environment using IDX. You can also run local emulators in your web-based development environment to test your application’s backend services.

For example, you can run the Firebase Local Emulator Suite directly from your IDX workspace. To install the emulator suite, you’d run firebase init emulators from the IDX Terminal tab and follow the steps to configure which emulators you want on what ports.

ALT TEXT

Once you’ve installed them, you can configure and use them the same way you would in a local development environment from the IDX terminal.


Next Steps

As you can see, Project IDX can meet many of your full-stack development needs — from frontend to backend and every emulator in between.

If you're already using Project IDX, tag us on social with #projectidx to let us know how Project IDX has helped you with your full-stack development. Or to sign up for the waitlist, visit idx.dev.

Global Developers Share How They Use Inclusive Design

Posted by Lyanne Alfaro, DevRel Program Manager, Google Developer Studio

Developer Journey is a monthly series highlighting diverse and global developers sharing relatable challenges, opportunities, and wins in their journey. Every month, we will spotlight developers around the world, the Google tools they leverage, and the kind of products they are building.

This month we speak with global developers across Google Developer Experts, Google Developer Groups, and beyond to learn more about their favorite Google tools, the applications they’ve built to serve diverse communities and the role of inclusive design in their process.


Lamis Chebbi

Headshot of Lamis Chebbi, smiling
Republic of Tunisia
Senior Software Engineer

What Google tools have you used to build?

I use Lighthouse and Google PageSpeed Insights to audit my application's performance and check my accessibility score. I can learn a lot about my application users and measure their engagement through Google Analytics. I have also used: Angular, Angular Dev tools, Firebase, TensorFlow and some services through Google Cloud Platform.


Which tool has been your favorite to use? Why?

On a daily basis I use Angular to develop my web applications. It helped me develop web applications faster with less code, less debugging time, and high scalability. The Angular CLI automates a lot of tasks, including the upgrade process, which saves a lot of time.


Please share with us about something you’ve built in the past using Google tools.

I have built a lot of web apps and progressive web apps using Angular, Firebase and TensorFlow in various fields from insurance, to banking, retail and education.


What will you create with Google Bard?

I'm planning on creating a blog using Google Bard and to generate content in different languages and enable some search and updates for content.


What role does inclusive design play in your development process?

Accessibility is no longer an option today. It is as important as other development goals and should be automated in the development process using the right tools.


What advice would you give someone starting in their developer journey?

Here’s a few pieces of advice for other professionals:

  • Invest in learning as much as you can and always practice the technologies you learn.
  • Don't forget that practice makes perfect.
  • Join developer communities and get a mentor; you will learn a lot and receive a lot of help.
  • Try to keep up with new technologies and trends that will open new perspectives for you.

You’ll probably make some mistakes. Be willing to accept it and learn from it.


Amani Bisimwa

Headshot of Amani Bisimwa, smiling
Bukavu, Democratic Republic of Congo
Google Developer Groups Uvira Lead
Frontend Developer

What Google tools have you used to build?

I am using Angular and Firebase.


Which tool has been your favorite to use? Why?

Firebase is my favorite. I like how Firebase has simplified things by providing a Backend as Service. You no longer need to manage your own servers, worry about scalability, or other Backend complexities.


Please share with us about something you’ve built in the past using Google tools.

I have built some private ERP apps that help small local traders to manage their business (stock management, finance and hotels).


What will you create with Google Bard?

I always use Bard for guidance to document and test code. I hope to use it for more projects in the future.


What role does inclusive design play in your development process?

The role of a designer in the development process is so important to me. Not only does it allow me to arrange the elements well on the screen, but it also ensures that the application is accessible to users living with disabilities. The designer also knows how to choose colors, contrasts and hierarchy of different elements.


How do you prioritize accessibility alongside other development goals?

Accessibility is a priority for me when creating an app or product. I consider accessibility at every stage of the development process. I use a variety of tools and resources to ensure my apps are accessible to everyone, including people with visual impairments, hearing impairments, motor disabilities and cognitive disabilities.


What advice would you give someone starting in their developer journey?

My advice is: Choose your path and stick to it because there are several distractions from the trends of new technologies on social media especially on Twitter. Don't skip the steps; learn the fundamentals.  It's important because even to improve a prompt with generative AI, you need to have a solid understanding in your field.



Enrique López Mañas

Headshot of Enrique López Mañas, smiling
Munich, Germany
Freelance Software Engineer

What Google tools have you used to build?

Android Studio is my daily tool. I have used other tools or frameworks (like Firebase or TensorFlow) in the past as well. My choice of tool depends on the needs of the project I am currently engaged with.


Which tool has been your favorite to use? Why?

Android Studio is my absolute favorite, which is not a surprise for an Android Developer.


Please share with us about something you’ve built in the past using Google tools.

I have worked in many apps and frameworks in the past. The Deutsche Bahn (German Train) application, a Corona app for the Arab Emirates, the app for Alibaba couriers in Vietnam, and now the Google Maps library for Compose.


What will you create with Google Bard?

Bard and other tools like ChatGPT help me with the development of apps and software in general. I do feel they are not yet ready to significantly impact the development process. They still suffer from many inaccuracies and hallucinations.


How do you prioritize accessibility alongside other development goals?

Much less than I would actually like to. Often companies are on a budget and some important things tend to get deprioritized. As a developer (and consultant) my role is to advise them, and A11y is one of the main topics that tend to be underrated.

For instance, do you know that approximately 20% of the users in Switzerland have some form of disability, and can benefit from apps with accessibility integrated? I was really surprised when I heard this number, and I am fairly confident most people don't know about it. If there were more awareness, apps would benefit more from A11y practices.


What advice would you give someone starting in their developer journey?

For new developers: ask all the questions. Never leave a room with a doubt or a question and without an answer. Even more senior people do not have all the answers all the time, and the only way to know if they do is to ask questions. Do not feel embarrassed by raising your hand in a meeting. Ask all the questions you need. The quality of your life will be determined by the quality of your questions.

Save the date for Firebase’s first Demo Day!

Posted by Annum Munir, Product Marketing Manager

This article was originally posted on the Firebase blog.

For the past six years, we have shared the latest and greatest updates to Firebase, Google’s app development platform, at our annual Firebase Summit – this year, we wanted to do something a little different for our community of developers. So, in addition to the Flutter Firebase festival that just wrapped up, and meeting you all over the world at DevFests, we’re thrilled to announce our very first Firebase Demo Day, happening on November 8, 2023!

What is Demo Day?

Demo Day will be a virtual experience where we'll unveil short demos (i.e. pre-recorded videos) that showcase what's new, what's possible, and how you can solve your biggest app development challenges with Firebase. You’ll hear directly from our team about what they’ve been working on in a format that will feel both refreshing but also familiar.

What will you learn?

You’ll learn how Firebase can help you build and run fullstack apps faster, harness the power of AI to build smart experiences, and use Google technology and tools together to be more productive. We’ve been working closely with our friends from Flutter, Google Cloud, and Project IDX to ensure the demos cover a variety of topics and feature integrated solutions from your favorite Google products.

How can you participate?

Since Demo Day is not your typical physical or virtual event, you don’t need to worry about registering, securing a ticket, or even traveling. This is one of the easiest ways to peek at the exciting future of Firebase! Simply bookmark the website (and add the event to your calendar), then check back on Wednesday, November 8, 2023 at 1:00 pm EST to watch the videos at your own pace and be inspired to make your app the best it can be for users and your business.

In the meantime, we encourage you to follow us on X (formerly Twitter) and LinkedIn and join the conversation using #FirebaseDemoDay. We’ll be sharing teasers and behind-the-scenes footage throughout October as we count down to Demo Day, so stay tuned!

Developers Share How They Build with Google Tools and Bard

Posted by Lyanne Alfaro, DevRel Program Manager, Google Developer Studio

Developer Journey is a monthly series highlighting diverse and global developers sharing relatable challenges, opportunities, and wins in their journey. Every month, we will spotlight developers around the world, the Google tools they leverage, and the kind of products they are building.

This month, we spoke with several Google Developer Experts to learn more about their path.


Eslam Medhat Fathy

Headshot of Eslam Medhat Fathy smiling
Giza, Egypt
Google Developer Expert, Firebase
Technical and Design Mentor at Google for Startups Accelerator Program
Google Developer Group Organizer
Senior Flutter Developer at Sarmad

What Google tools have you used to build?

I have used many tools like Firebase, Flutter, Android, Kotlin, Dart, Assistant, and Bard, of course.

Which tool has been your favorite to use? Why?

My favorite tool is Firebase, because of how easy it is to set up and use. It also provides a serverless architecture, easy-to-use services, real-time synchronization, and cross-platform support, among other features. These benefits can help you build robust and scalable applications quickly and easily.

Tell us about something you've built in the past using Google tools.

I have more than 10 apps in the store created in Android native with Kotlin, Flutter and Dart. A few examples are Rehlatech and AzkarApp.

What will you create with Google Bard?

I use Bard every day for generating, debugging, explaining, learning code, and more.

What advice would you give someone starting in their developer journey?

I advise everyone about to start their developer journey to:

  • Start with the basics: It's important to have a solid foundation in programming fundamentals. Learn the basics of a programming language, such as syntax, data types, control structures, and functions.
  • Practice coding: Practice makes perfect. The more you practice coding, the better you'll become. Start with small projects and gradually move on to more complex projects.
  • Learn from others: Join online communities, attend meetups, and participate in forums. Learning from others can help you improve your skills.
  • Read the documentation: Documentation is your friend. Read the documentation of the programming language or tools you're using. It can help you understand how to use them properly and solve problems.
  • Be patient: Learning to code takes time and patience. Don't get discouraged if you don't understand something right away. Keep practicing and asking questions.
  • Build projects: Building projects is a great way to learn new skills and apply what you've learned. Start small and gradually build more complex projects.
  • Stay up-to-date: Technology is constantly evolving. Stay up-to-date on the latest trends and updates in the programming world. Attend conferences, read blogs, and follow experts on social media.
  • Have fun: Coding should be fun. Don't take it too seriously and enjoy the process of learning and building new things.

Carmen Ansio

Headshot of Carmen Ansio smiling
Barcelona, Spain
Google Developer Expert, Firebase
Google Developer Expert, Web Technologies
UX Engineer

What Google tools have you used to build?

I have used various Google tools to build projects including Angular, Dart, and Firebase.

Which tool has been your favorite to use? Why?

My favorite tool has been Chrome DevTools because of its versatile suite of debugging tools and its network panel, which I often use to optimize web performance. DevTools is an essential part of my daily development process as it allows me to test, experiment, and debug code directly in the browser.

What will you create with Google Bard?

With Google Bard, I plan to develop a Figma plugin for creating dynamic design prototypes. Leveraging the natural language processing and understanding capabilities of Google Bard, the plugin will allow designers to quickly convert textual descriptions into visual design elements. This can significantly streamline the design process, bridging the gap between ideation and visual representation, while enabling non-designers to contribute effectively to the design process.

What advice would you give someone starting in their developer journey?

For those beginning their developer journey, my advice would be: Always stay curious and never stop learning. Technology evolves quickly, and it's important to be adaptable. Also, never undervalue the importance of good UI/UX design. It's not only about writing code, but also about creating a great user experience.


Stéphanie Walter

Headshot of Stéphanie Walter smiling
Luxembourg, Luxembourg
Google Developer Expert, Web Technologies
Women Techmakers
UX Researcher & Designer

What Google tools have you used to build?

The main tools I use are the Chrome inspect tool and Lighthouse. I’m using Material UI a lot and the M3 design kit for Figma is a great time saver.

Which tool has been your favorite to use? Why?

Performance is important where I work, so Lighthouse is definitely in my favorite list. The function to get a quick report, which also shows main accessibility issues, is very nice. Of course it won’t show all accessibility issues, but it’s a good place to start improving a website.

Please share with us about something you’ve built in the past using Google tools.

Both Lighthouse and the Chrome inspect tool are lifesavers when building websites like my blog. There’s still improvement to be made on some pages on performance, but it’s getting there.

What will you create with Google Bard?

To be honest, it only has been recently made available for my country, so I haven’t had time to really play with it. For now, I use AI chatbots as glorified assistants. English isn’t my native language, so asking such tools to help translate some things and improve grammar in some sentences is very helpful. I might use it to help me with sharing knowledge: to improve my articles, conference slides, and training material.

What advice would you give someone starting in their developer journey?

Start with a project you are passionate about, something that would help you, or something you wish existed. It doesn’t have to be perfect. It also doesn’t have to be something that will bring money. And remember, you also don’t have to finish it. It’s nice if you can share it with peers to get feedback but you can also share unfinished projects. It’s all about learning while working on something that you like.But remember to also step away from the computer. Developing should not be your whole life - otherwise, you will burn out really fast.

MediaPipe: Enhancing Virtual Humans to be more realistic

A guest post by the XR Development team at KDDI & Alpha-U

Please note that the information, uses, and applications expressed in the below post are solely those of our guest author, KDDI.

AI generated rendering of virtual human ‘Metako’
KDDI is integrating text-to-speech & Cloud Rendering to virtual human ‘Metako’

VTubers, or virtual YouTubers, are online entertainers who use a virtual avatar generated using computer graphics. This digital trend originated in Japan in the mid-2010s, and has become an international online phenomenon. A majority of VTubers are English and Japanese-speaking YouTubers or live streamers who use avatar designs.

KDDI, a telecommunications operator in Japan with over 40 million customers, wanted to experiment with various technologies built on its 5G network but found that getting accurate movements and human-like facial expressions in real-time was challenging.


Creating virtual humans in real-time

Announced at Google I/O 2023 in May, the MediaPipe Face Landmarker solution detects facial landmarks and outputs blendshape scores to render a 3D face model that matches the user. With the MediaPipe Face Landmarker solution, KDDI and the Google Partner Innovation team successfully brought realism to their avatars.


Technical Implementation

Using Mediapipe's powerful and efficient Python package, KDDI developers were able to detect the performer’s facial features and extract 52 blendshapes in real-time.

import mediapipe as mp from mediapipe.tasks import python as mp_python MP_TASK_FILE = "face_landmarker_with_blendshapes.task" class FaceMeshDetector: def __init__(self): with open(MP_TASK_FILE, mode="rb") as f: f_buffer = f.read() base_options = mp_python.BaseOptions(model_asset_buffer=f_buffer) options = mp_python.vision.FaceLandmarkerOptions( base_options=base_options, output_face_blendshapes=True, output_facial_transformation_matrixes=True, running_mode=mp.tasks.vision.RunningMode.LIVE_STREAM, num_faces=1, result_callback=self.mp_callback) self.model = mp_python.vision.FaceLandmarker.create_from_options( options) self.landmarks = None self.blendshapes = None self.latest_time_ms = 0 def mp_callback(self, mp_result, output_image, timestamp_ms: int): if len(mp_result.face_landmarks) >= 1 and len( mp_result.face_blendshapes) >= 1: self.landmarks = mp_result.face_landmarks[0] self.blendshapes = [b.score for b in mp_result.face_blendshapes[0]] def update(self, frame): t_ms = int(time.time() * 1000) if t_ms <= self.latest_time_ms: return frame_mp = mp.Image(image_format=mp.ImageFormat.SRGB, data=frame) self.model.detect_async(frame_mp, t_ms) self.latest_time_ms = t_ms def get_results(self): return self.landmarks, self.blendshapes

The Firebase Realtime Database stores a collection of 52 blendshape float values. Each row corresponds to a specific blendshape, listed in order.

_neutral, browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, ...

These blendshape values are continuously updated in real-time as the camera is open and the FaceMesh model is running. With each frame, the database reflects the latest blendshape values, capturing the dynamic changes in facial expressions as detected by the FaceMesh model.

Screenshot of realtime Database

After extracting the blendshapes data, the next step involves transmitting it to the Firebase Realtime Database. Leveraging this advanced database system ensures a seamless flow of real-time data to the clients, eliminating concerns about server scalability and enabling KDDI to focus on delivering a streamlined user experience.

import concurrent.futures import time import cv2 import firebase_admin import mediapipe as mp import numpy as np from firebase_admin import credentials, db pool = concurrent.futures.ThreadPoolExecutor(max_workers=4) cred = credentials.Certificate('your-certificate.json') firebase_admin.initialize_app( cred, { 'databaseURL': 'https://your-project.firebasedatabase.app/' }) ref = db.reference('projects/1234/blendshapes') def main(): facemesh_detector = FaceMeshDetector() cap = cv2.VideoCapture(0) while True: ret, frame = cap.read() facemesh_detector.update(frame) landmarks, blendshapes = facemesh_detector.get_results() if (landmarks is None) or (blendshapes is None): continue blendshapes_dict = {k: v for k, v in enumerate(blendshapes)} exe = pool.submit(ref.set, blendshapes_dict) cv2.imshow('frame', frame) if cv2.waitKey(1) & 0xFF == ord('q'): break cap.release() cv2.destroyAllWindows() exit()

 

To continue the progress, developers seamlessly transmit the blendshapes data from the Firebase Realtime Database to Google Cloud's Immersive Stream for XR instances in real-time. Google Cloud’s Immersive Stream for XR is a managed service that runs Unreal Engine project in the cloud, renders and streams immersive photorealistic 3D and Augmented Reality (AR) experiences to smartphones and browsers in real time.

This integration enables KDDI to drive character face animation and achieve real-time streaming of facial animation with minimal latency, ensuring an immersive user experience.

Illustrative example of how KDDI transmits data from the Firebase Realtime Database to Google Cloud Immersive Stream for XR in real time to render and stream photorealistic 3D and AR experiences like character face animation with minimal latency

On the Unreal Engine side running by the Immersive Stream for XR, we use the Firebase C++ SDK to seamlessly receive data from the Firebase. By establishing a database listener, we can instantly retrieve blendshape values as soon as updates occur in the Firebase Realtime database table. This integration allows for real-time access to the latest blendshape data, enabling dynamic and responsive facial animation in Unreal Engine projects.

Screenshot of Modify Curve node in use in Unreal Engine

After retrieving blendshape values from the Firebase SDK, we can drive the face animation in Unreal Engine by using the "Modify Curve" node in the animation blueprint. Each blendshape value is assigned to the character individually on every frame, allowing for precise and real-time control over the character's facial expressions.

Flowchart demonstrating how BlendshapesReceiver handles the database connection, authentication, and continuous data reception

An effective approach for implementing a realtime database listener in Unreal Engine is to utilize the GameInstance Subsystem, which serves as an alternative singleton pattern. This allows for the creation of a dedicated BlendshapesReceiver instance responsible for handling the database connection, authentication, and continuous data reception in the background.

By leveraging the GameInstance Subsystem, the BlendshapesReceiver instance can be instantiated and maintained throughout the lifespan of the game session. This ensures a persistent database connection while the animation blueprint reads and drives the face animation using the received blendshape data.

Using just a local PC running MediaPipe, KDDI succeeded in capturing the real performer’s facial expression and movement, and created high-quality 3D re-target animation in real time.

Flow chart showing how a real performer's facial expression and movement being captured and run through MediaPipe on a Local PC, and the high quality 3D re-target animation being rendered in real time by KDDI
      

KDDI is collaborating with developers of Metaverse anime fashion like Adastria Co., Ltd.


Getting started

To learn more, watch Google I/O 2023 sessions: Easy on-device ML with MediaPipe, Supercharge your web app with machine learning and MediaPipe, What's new in machine learning, and check out the official documentation over on developers.google.com/mediapipe.


What’s next?

This MediaPipe integration is one example of how KDDI is eliminating the boundary between the real and virtual worlds, allowing users to enjoy everyday experiences such as attending live music performances, enjoying art, having conversations with friends, and shopping―anytime, anywhere. 

KDDI’s αU provides services for the Web3 era, including the metaverse, live streaming, and virtual shopping, shaping an ecosystem where anyone can become a creator, supporting the new generation of users who effortlessly move between the real and virtual worlds.

Experts share insights on Firebase, Flutter and the developer community

Posted by Komal Sandhu - Global Program Manager, Google Developer Groups

Rich Hyndman, Manager, Firebase DevRel (left) and Eric Windmill, Developer Relations Engineer, Firebase and Flutter (right)

Firebase and Flutter offer many tools that ‘just work’, which is something that all apps need. I think you’d be hard pressed to find another combination of front end framework and back end services that let developers make apps quickly without sacrificing quality.” 

moving images of Sparky and Dart, respective mascots for Firebase and Flutter
Among the many inspiring experts in the developer communities for Firebase and Flutter are Rich Hyndman and Eric Windmill. Each Googler serves their respective product team from the engineering and community sides and has a keen eye towards the future. Read on to see their outlook on their favorite Firebase and Flutter tools and the developers that inspire them.

===

What is your title, and how long have you been at Google?

Rich: I run Firebase Developer Relations,, I’ve been at Google for around 11 years

Eric: I’m an engineer on the Flutter team and I’ve been at Google for a year.


Tell us about yourself:

Rich: I’ve always loved tech, from techy toys as a kid to anything that flies. I still get tech-joy when I see new gadgets and devices. I built and raced drones for a while, but mobile/cell phones are the ultimate gadget for me and enabled my career.

Eric: I’m a software engineer, and these days I’m specifically a Developer Relations Engineer. I’m not surprised I’ve ended up here, as I like to joke “I like computers but I like people more.” Outside of work, most of my time is spent thinking about music. I’m pretty poor at playing music, but I’ve always consumed as much as I could. If I had to choose a different job and start over, I’d be a music journalist.


How did you get started in this space?

Rich: I've always loved mobile apps: being able to carry my work in my pocket, play with it, test it, demo it, and be proud of it. From the beginning of my career right up till today, it's still the best. I worked on a few mobile projects pre-Android and was part of an exciting mobile tech startup for a few years, but it was Android that really kick-started my career.

I quickly fell in love with the little green droid and the entire platform, and through a combination of meetups, competition entries and conferences I ended up in contact with Android DevRel at Google.

Firebase is a natural counterpart to Android and I love being able to support developers from a different angle. Firebase also supports Flutter, Web and iOS, Firebase, which has also given me the opportunity to learn more about other platforms and meet more developers.

Eric: I got into this space by accident. At my first software job, the company was already using Dart for their web application, and started rebuilding their mobile apps in Flutter soon after I joined. I think that was around 2016 or 2017. Flutter was still in its Alpha stage. I was introduced to Firebase at the same job, and I’ve used various tools from the Firebase SDK ever since.


What are some challenges that you have seen developers being facing?

Rich: Developers often want to get up and running with new projects quickly, but then iterate and improve their apps. No-code solutions can be great to start with but aren’t flexible enough down the road. A lower-code solution like Firebase can be quick to get started, and it can also provide control. Bringing Flutter and Firebase together creates a powerful and flexible combination.

Eric: Regardless of the technology, I think the biggest challenge developers face is actually with documentation. It doesn’t matter how good a product is if the docs are hard to find or hard to understand. We’ve seen this ourselves recently as Flutter became an “official” supported platform on Firebase in May 2022. When that happened, we moved the documentation from the Flutter site to the Firebase site, and folks didn’t know how to find the docs. It was an oversight on our part, but it’s a good example of the importance of docs. They deserve way more attention than they get in many, many cases.

image of Sparky and Dart, respective mascots for Firebase and Flutter
What do you think is the most interesting or useful resource to learn more about Firebase & Flutter? Is there a particular library or codelab that everyone should learn?

Rich: The official docs have to be first, located at firebase.google.com. We have a great repository of Learning Pathways, including Add Firebase to your Flutter App. We’re also just launching our new Solutions Portal with over 60 solutions guides indexed already.

Eric: If I have to name only one resource, it’d be this codelab: Get to know Firebase for Flutter
But Firebase offers so many tools. This codelab is just an introduction to what’s possible.


What are some inspiring ways that developers are building together Firebase and Flutter?

Rich: We’ve had an interesting couple of years at Firebase. Firebase has always been known for powering real-time data driven apps. If you used a Covid stats app during the pandemic there’s a fair chance it was running on Firebase; there was a big surge of new apps.

Eric: Lately I’ve seen an interest in using Flutter to make 2D games, and using some Firebase tools for the back end of the game. I love this. Games are just more fun than apps, of course, but it’s also great to see folks using these technologies in ways that aren’t the explicit purposes. It shows creativity and excellent problem solving.


What’s a specific use case of Firebase & Flutter technology that excites you?

Rich: Firebase Extensions are very exciting. They are pre-packaged bundles of code that make it easy to add new features to your app from Google and partners like Stripe and Vonage. We just launched the Extensions Marketplace and opened up the ability for developers to build extensions for their own apps through our Provider Alpha program.

Eric: Flutter web and Firebase hosting is just a no brainer. You can deploy a Flutter app to the web in no time.


How can developers be successful building on Firebase & Flutter?

Rich: There’s a very powerful combination with Crashlytics, Performance Monitoring, A/B Testing and Remote Config. Developers can quickly improve the stability of their apps whilst also iterating on features to deliver the best experience for their users. We’ve had a lot of success with improving monetization, too. Check out some of our case studies for more details.

Eric: Flutter developers can be successful by leveraging all that Firebase offers. Firebase might seem intimidating because it offers so much, but it excels at being easy to use, and I encourage all web and mobile developers to poke around. They’re likely to find something that makes their lives easier.

image of Firebase and Flutter logos against a dot matrix background
What’s next for the Firebase & Flutter Communities? What might the future look like?

Rich: Over the next year we’ll be focusing on modern app development and some more opinionated guides. Better support for Flutter, Kotlin, Jetpack Compose, Swift/SwiftUI and modern web frameworks.

Eric: There is a genuine effort amongst both teams to support each other. Flutter and Firebase are just such a great pair, that it makes sense for us to encourage our communities to check out one another. In the future, I think this will continue. I think you’ll see a lot of Flutter at Firebase events, and vice versa.


How does Firebase & Flutter help expand the impact of developers?

Rich: Firebase has always focused on helping developers get their apps up and running by providing tools to streamline time-consuming tasks. Enabling developers to focus on delivering the best app experiences and the most value to their users.

Eric: Flutter is an app-building SDK that is a joy to use. It seriously increases velocity because it’s cross-platform. Firebase and Flutter offer many tools that “just work”, which is something that all apps need. I think you’d be hard pressed to find another combination of front end framework and back end services that let developers make apps quickly without sacrificing quality.


Find a Google Developer Group hosting a DevFest near you.

Want to learn more about Google Technologies like Firebase & Flutter? Hoping to attend a DevFest or Google Developer Groups (GDG)? Find a GDG hosting a DevFest near you here.