Tag Archives: Storage

Updating Google Photos’ storage policy to build for the future

We launched Google Photos more than five years ago with the mission of being the home for your memories. What started as an app to manage your photos and videos has evolved into a place to reflect on meaningful moments in your life. Today, more than 4 trillion photos are stored in Google Photos, and every week 28 billion new photos and videos are uploaded. 

Since so many of you rely on Google Photos to store your memories, it’s important that it’s not just a great product, but also continues to meet your needs over the long haul. In order to welcome even more of your memories and build Google Photos for the future, we are changing our unlimited High quality storage policy. 

Starting June 1, 2021, any new photos and videos you upload will count toward the free 15 GB of storage that comes with every Google Account or the additional storage you’ve purchased as a Google One member. Your Google Account storage is shared across Drive, Gmail and Photos. This change also allows us to keep pace with the growing demand for storage. And, as always, we uphold our commitment to not use information in Google Photos for advertising purposes. We know this is a big shift and may come as a surprise, so we wanted to let you know well in advance and give you resources to make this easier. 


Existing High quality photos and videos are exempt from this change 


Any photos or videos you’ve uploaded in High quality before June 1, 2021 will not count toward your 15GB of free storage. This means that photos and videos backed up before June 1, 2021 will still be considered free and exempt from the storage limit. You can verify your backup quality at any time in the Photos app by going to back up & sync in Settings.


If you back up your photos and videos in Original quality, these changes do not affect you. As always, your Original quality photos and videos will continue to count toward your 15 GB of free storage across your Google Account. 

If you have a Pixel 1-5, photos uploaded from that device won’t be impacted. Photos and videos uploaded in High quality from that device will continue to be exempt from this change, even after June 1, 2021. 


There’s no action you need to take today


This change does not take effect for another six months, so you don’t need to do anything right now. And once this change does take effect on June 1, 2021, over 80 percent of you should still be able to store roughly three more years worth of memories with your free 15 GB of storage. As your storage nears 15 GB, we will notify you in the app and follow up by email. 


Understand and manage your quota


To understand how this impacts you, you can see a personalized estimate for how long your storage may last. This estimate takes into account how frequently you back up photos, videos and other content to your Google Account.


And in June 2021, you’ll be able to access a new free tool in the Photos app to easily manage your backed up photos and videos. This tool will help you review the memories you want to keep while also surfacing shots you might prefer to delete, like dark or blurry photos or large videos.


If you decide you want more space, you can always expand your storage through Google One. Plans start at $1.99 per month in the U.S. for 100 GB of storage and include additional member benefits like access to Google experts, shared family plans and more.

 

Thank you for using Google Photos and we hope to continue to be the home for your memories. You can learn more about this change in our Help Center.


Posted by Shimrit Ben-Yair, Vice President, Google Photos


An update to storage policies across your Google Account

Over the past decade, Gmail, Google Drive and Google Photos have helped billions of people securely store and manage their emails, documents, photos, videos and more. Today, people are uploading more content than ever before—in fact, more than 4.3 million GB are added across Gmail, Drive and Photos every day. 

To continue providing everyone with a great storage experience and to keep pace with the growing demand, we're announcing important upcoming storage changes to your Google Account. These changes will apply to Photos and Drive (specifically Google Docs, Sheets, Slides, Drawings, Forms and Jamboard files) and will enable us to continue investing in these products for the future. We're also introducing new policies for consumer Google Accounts that are either inactive or over their storage limit across Gmail, Drive (including Google Docs, Sheets, Slides, Drawings, Forms and Jamboard files) and Photos, to bring our policies more in line with industry standards. 

These storage policy changes won’t take effect until June 1, 2021. However, we wanted to let you know well in advance and give you the resources to navigate these changes. Google Workspace subscribers, and G Suite for Education and G Suite for Nonprofits customers should refer to our Google Workspace Updates post to understand how these changes may affect them.

As always, every Google Account will continue to come with 15 GB of free storage across Gmail, Drive and Photos, which we estimate should last the majority of our users several years.  Because the content you store with these apps is primarily personal, it’s not used for advertising purposes. We’ll also continue to give you visibility and control over your storage, and provide tools to help you easily manage it. 


New content that will count toward your Google Account storage


Beginning June 1, any new photo or video uploaded in High quality in Google Photos will count toward your free 15 GB storage quota or any additional storage you’ve purchased as a Google One member. To make this transition easier, we’ll exempt all High quality photos and videos you back up before June 1. This includes all of the High quality photos and videos you currently store with Google Photos. Most people who back up in High quality should have years before they need to take action—in fact, we estimate that 80 percent of you should have at least three years before you reach 15 GB. You can learn more about this change in our Google Photos post.


Also starting June 1, any new Docs, Sheets, Slides, Drawings, Forms or Jamboard file will begin counting toward your free 15 GB of allotted storage or any additional storage provided through Google One. Existing files within these products will not count toward storage, unless they’re modified on or after June 1. You can learn more in our Help Center.


A new policy for accounts that are inactive or over storage limit


We’re introducing new policies for consumer accounts that are either inactive or over their storage limit across Gmail, Drive (including Google Docs, Sheets, Slides, Drawings, Forms and Jamboard files) and/or Photos to better align with common practices across the industry. After June 1: 

  • If you're inactive in one or more of these services for two years (24 months), Google may delete the content in the product(s) in which you're inactive. 

  • Similarly, if you're over your storage limit for two years, Google may delete your content across Gmail, Drive and Photos.

We will notify you multiple times before we attempt to remove any content so you have ample opportunities to take action. The simplest way to keep your account active is to periodically visit Gmail, Drive or Photos on the web or mobile, while signed in and connected to the internet. 

The Inactive Account Manager can help you manage specific content and notify a trusted contact if you stop using your Google Account for a certain period of time (between 3-18 months). Note that the new two year inactive policy will apply regardless of your Inactive Account Manager settings. 

You can learn more about these changes in our Help Center.


How to manage your storage


To help you manage your Google Account storage, anyone can use the free storage manager in the Google One app and on the web, which gives you an easy way to see how you’re using your storage across Gmail, Drive and Photos. You can keep the files you want, delete the ones you no longer need and make room for more—all in one place.



In addition to helping us meet the growing demand for storage, these changes align our storage policies across products. As always, we remain committed to providing you a great experience and hope to continue to serve you in the future. You can learn more about this change in our Help Center.


Posted by Jose Pastor, Vice President, Google Workspace, and Shimrit Ben-Yair, Vice President, Google Photos


Android Q Beta 5 Update

Posted by Dave Burke, VP of Engineering

AndroidQ logo

Android Q Beta 5 launches today! Today we're rolling out Beta 5, bringing Android Q Beta very close to the system behaviors you'll see in the final release. Developer APIs were already finalized in the previous update. So, now is the time to test your apps for compatibility and make sure they are ready!

You can get Beta 5 today on Pixel devices by enrolling here. If you're already enrolled and received Beta 4 on your Pixel device, you'll automatically get the update to Beta 5. Partners participating in the Android Q Beta program will also be updating their devices to Beta 5 over the coming weeks.

To get started with Android Q Beta, visit developer.android.com/preview.

What’s in Beta 5?

The Beta 5 update includes the latest Android Q system images for Pixel and Android jEmulator, along with the final Android Q developer APIs (API level 29), the official API 29 SDK, and updated build tools for Android Studio. These give you everything you need to test your apps on Android Q and build with Android Q features.

Gestural navigation updates

As we talked about at Google I/O, we’ve been working closely with device-maker partners to ensure a standardized Android gestural navigation for users and developers. Gestural navigation lets apps use the full screen for content while minimizing the visible system chrome and navigation – which is particularly important on today’s edge-to-edge screens. In Beta 5 we’re continuing to improve and polish based on your feedback and we wanted to provide an update on a few key areas.

We’ve introduced a swipe gesture from either corner to get to the Assistant - you’ll notice indicators in the bottom corners that we’re continuing to tune.

For apps using a navigation drawer, we’ve added a peek behavior when users have grabbed the drawer to indicate that a swipe will bring in the navigation drawer. This works for all versions of DrawerLayout, with DrawerLayout 1.1.0-alpha02 optimized for the best experience.

Custom launchers are another area where we’ve heard feedback and we’re continuing to work on issues, particularly with stability and Recents. Starting in Beta 6, we’ll switch users to 3-button navigation when they are using a custom launcher by default. We’ll address the remaining issues in a post-launch update allowing all users to switch to gestural navigation. Meanwhile, please continue to give us your feedback.

Get your apps ready for Android Q!

With the consumer release coming soon, it’s highest priority for all Android developers to update your current apps for compatibility as soon as possible.

Here’s how to do it:

We realize that supporting these changes is an investment for you too, and we're working to minimize the impact on your apps and be responsive to your input as we move toward the final release.

Enhance your app with Android Q features and APIs

Next, when you're ready, dive into Android Q and learn about the new features and APIs that you can use. Here are some of the top features to get started with.

We recommend these for every app:

  • Dark Theme: Ensure a consistent experience for users who enable system-wide dark theme by adding a Dark Theme or enabling Force Dark.
  • Support gestural navigation in your app by going edge-to-edge and making sure your custom gestures are complementary to the system navigation gestures.
  • Optimize for foldables: Deliver seamless, edge-to-edge experiences on today’s innovative devices by optimizing for foldables.

We recommend these if relevant for your app:

  • More interactive notifications: If your notifications include messages, enable suggested replies and actions in notifications to engage users and let them take action instantly.
  • Better biometrics: If you use biometric auth, move to BiometricPrompt, the preferred way to support fingerprint auth on modern devices.
  • Enriched recording: To support captioning or gameplay recording, enable audio playback capture -- it’s a great way to reach more users and make your app more accessible.
  • Better codecs: For media apps, try AV1 for video streaming and HDR10+ for high dynamic range video. For speech and music streaming, you can use Opus encoding, and for musicians, a native MIDI API is available.
  • Better networking APIs: If your app manages IoT devices over Wi-Fi, try the new network connection APIs for functions like configuring, downloading, or printing.

These are just a few of the many new features and APIs in Android Q -- to see them all, visit the Android Q Beta site for developers.

Publish your app updates to Google Play

As soon as you're ready, publish your APK updates to Google Play that are compiled against, or optionally targeting, API 29. To make sure that your updated app runs well on Android Q as well as older versions, try using Google Play testing tracks. With tracks you can safely get early feedback from a small group of users -- including Beta 5 users — and then do a staged rollout to production.

How do I get Beta 5?

It’s easy! Just enroll any supported Pixel device here to get the update over-the-air. If you're already enrolled, you'll receive the update soon and no action is needed on your part. Downloadable system images are also available here. Partners who are participating in the Android Q Beta program will be updating their devices over the coming weeks. See android.com/beta for details.

To get started developing, download the official API 29 SDK and tools into the stable release of Android Studio 3.4, or for the latest Android Q support update to Android Studio 3.5 Beta. Then follow these instructions to configure your environment, and see the release notes for known issues.

There will be one more Beta release before the consumer launch later this quarter. Please continue to share your feedback and requests -- you can use our hotlists for filing platform issues (including privacy and behavior changes), app compatibility issues, and third-party SDK issues.

Also, the Android engineering team will host a Reddit AMA on r/androiddev to answer your technical questions about Android Q later this month. Look out for an announcement on r/androiddev with details in the coming weeks. We look forward to addressing your questions!

Android Q Beta 4 and Final APIs!

Posted by Dave Burke, VP of Engineering

AndroidQ logo

Last month at Google I/O we talked about what’s new for Android developers, from new features in Android Q to the latest in Kotlin and Jetpack.

With Android Q, we highlighted three themes: innovation, security and privacy, and digital wellbeing. We want to help you take advantage of the latest new technology -- 5G, foldables, edge-to-edge screens, on-device machine learning, and more -- while making sure users' security, privacy, and wellbeing are always a top priority.

We also talked about how we’re going increasingly Kotlin-first, and continuing to expand Jetpack with new libraries like CameraX, Jetpack Security and Jetpack Compose -- a modern reactive-style UI toolkit for Android that takes advantage of Kotlin. If you missed the livestream for the keynotes or tech sessions, make sure to check out the full playlist of Android and Play sessions.

Today we’re releasing Beta 4 with the final Android Q APIs and official SDK -- the time is now to get your apps ready for the final release later in the summer!

You can get Beta 4 today on Pixel devices by enrolling here. If you're already enrolled and received the Beta 3 on your Pixel device, you'll automatically get the update to Beta 4. Partners participating in the Android Q Beta program will also be updating their devices to Beta 4 over the coming weeks.

To get started with Android Q Beta, visit developer.android.com/preview.

What’s in Beta 4?

The Beta 4 update includes the latest Android Q system images for Pixel and Android Emulator, along with the final Android Q developer APIs (API level 29), the official API 29 SDK, and updated build tools for Android Studio. Together, these give you everything you need to test your apps for compatibility with Android Q and build with Android Q features and APIs.

To get started, download the official API 29 SDK and tools into the stable release of Android Studio 3.4, or for the latest Android Q support update to Android Studio 3.5 Beta. Then follow these instructions to configure your environment, and see the release notes for known issues.

Make your apps compatible with Android Q!

With the developer APIs finalized and release candidate builds coming soon, it’s critical for all Android developers to test their current apps for compatibility with Android Q. We recommend getting started as soon as possible.

Just install your current app from Google Play onto an Android Q Beta device or emulator, then test. As you work through the flows, your app should run and look great and handle all of the Android Q behavior changes properly. Watch for impacts from privacy changes, gestural navigation, changes to dynamic linker paths for Bionic libraries, and others.

Make sure that you test with the Android Q privacy features, such as the new location permissions, restrictions on background activity starts, changes to data and identifiers, and other key privacy features. See the privacy checklist to get started, and review the behavior changes doc for more areas to test.

Android Developers YouTube channel UI on landscape mode.

You can use the updated Android Emulator to test your apps for compatibility.

If you plan to update your platform targeting to API 29, also make sure to test with scoped storage, location permission for wireless scans, and permission for fullscreen intents. You can read about other changes that could affect apps here.

It's also important to test for uses of restricted non-SDK interfaces and move to public SDK or NDK equivalents instead. Watch for logcat warnings that highlight these accesses and use the StrictMode method detectNonSdkApiUsage() to catch them programmatically.

Last, make sure to fully test the libraries and SDKs in your app to make sure they work as expected on Android Q and follow best practices for privacy, performance, UX, data handling, and permissions. If you find an issue, try updating to the latest version of the SDK, or reach out to the SDK developer for help. You can also report SDK compatibility issues here.

When you’ve finished your testing and made any updates, we recommend publishing your compatible app right away. This lets Android Beta users test the app now, and helps you deliver a smooth transition to users as they update to Android Q.

We realize that supporting these changes is an investment for you too, and we're working to minimize the impact on your apps and be responsive to your input as we move toward the final release in the coming months.

Enhance your app with Android Q features and APIs

When you're ready, dive into Android Q and learn about the new features and APIs that you can use in your apps. Android Q features can help you engage users, give them more control and security, and even improve your app's performance.

mobile device notification window

Android Q provides system-suggested replies and actions in notifications.

For example, you can deliver seamless, edge-to-edge experiences on today’s innovative devices by optimizing for foldables and supporting gestural navigation in your app. To engage more users, try supporting Dark Theme, suggested replies and actions in notifications, sharing shortcuts, and settings panels.

Google Maps app closes to display home screen with ocean aerial image

Gestural navigation lets you offer an edge-to-edge experience in your apps.

If your app manages IoT devices over Wi-Fi, try the new network connection APIs for functions like configuring, downloading, or printing. If your app manages Wi-Fi internet connections, try the network suggestion APIs as an easier way to surface preferred Wi-Fi networks, without needing to request location permission.

If you use the camera, learn about dynamic depth format. For media, you can use AV1 for video streaming and HDR10+ for high dynamic range video. For speech and music streaming, you can use Opus encoding, and for musicians, a native MIDI API is available.

Dynamic Depth lets you offer specialized blurs and bokeh options in your app.

To support captioning or gameplay recording, enable audio playback capture -- it’s a great way to reach more users and get your app noticed. If your app uses power intensively, try using the new thermal API to optimize app performance based on device temperature.

BiometricPrompt is now the preferred way to support fingerprint auth on modern devices, so all developers using fingerprint or other biometric auth should move to using this API as soon as possible. To make the transition easy, use the backwards-compatible BiometricPrompt API that we’re providing in the AndroidX library. Android Q supports both standard and passive (no confirmation, for face and other passive modes) auth flows.

These are just a few of the many new features and APIs in Android Q -- to see them all, visit the Android Q Beta site for developers.

Publish your app updates to Google Play

Today with Android Q Beta 4 we’re also opening up publishing on Google Play to apps that are compiled against, or optionally targeting, API 29. This means you can now push your updates to users now through Google Play to test your app’s compatibility, including on devices running Android Q Beta 4.

How do I get Beta 4?

It’s easy! Just enroll any supported Pixel device here to get the update over-the-air. If you're already enrolled, you'll receive the update soon and no action is needed on your part. Downloadable system images are also available here. Partners that are participating in the Android Q Beta program will be updating their devices over the coming weeks. See android.com/beta for details.

For even broader testing on supported devices, you can also get Android GSI images, and if you don’t have a device you can test on the Android Emulator.

As always, your input is critical, so please continue to let us know what you think. You can use our hotlists for filing platform issues (including privacy and behavior changes), app compatibility issues, and third-party SDK issues. You've shared great feedback with us so far and we're working to integrate as much of it as possible in the next Beta release.

We're looking forward to seeing your apps on Android Q!

What’s New in Android: Q Beta 3 & More

Posted by Dave Burke, VP, Engineering

Today Android is celebrating two amazing milestones. It’s Android’s version 10! And today, Android is running on more than 2.5B active Android devices.

With Android Q, we’ve focused on three themes: innovation, security and privacy, and digital wellbeing. We want to help you take advantage of the latest new technology -- 5G, foldables, edge-to-edge screens, on-device AI, and more -- while making sure users' security, privacy, and wellbeing are always a top priority.

Earlier at Google I/O we highlighted what’s new in Android Q and unveiled the latest update, Android Q Beta 3. Your feedback continues to be extremely valuable in shaping today’s update as well as our final release to the ecosystem in the fall.

This year, Android Q Beta 3 is available on 15 partner devices from 12 OEMs -- that’s twice as many devices as last year! It’s all thanks to Project Treble and especially to our partners who are committed to accelerating updates to Android users globally -- Huawei, Xiaomi, Nokia, Sony, Vivo, OPPO, OnePlus, ASUS, LGE, TECNO, Essential, and realme.

Visit android.com/beta to see the full list of Beta devices and learn how to get today’s update on your device. If you have a Pixel device, you can enroll here to get Beta 3 -- if you’re already enrolled, watch for the update coming soon. To get started developing with Android Q Beta, visit developer.android.com/preview.

Privacy and security

As we talked about at Google I/O, privacy and security are important to our whole company and in Android Q we’ve added many more protections for users.

Privacy

In Android Q, privacy has been a central focus, from strengthening protections in the platform to designing new features with privacy in mind. It’s more important than ever to give users control -- and transparency -- over how information is collected and used by apps, and by our phones.

Building on our work in previous releases, Android Q includes extensive changes across the platform to improve privacy and give users control -- from improved system UI to stricter permissions to restrictions on what data apps can use.

For example, Android Q gives users more control over when apps can get location. Apps still ask the user for permission, but now in Android Q the user has greater choice over when to allow access to location -- such as only while the app is in use, all the time, or never. Read the developer guide for details on how to adapt your app for the new location controls.

Outside of location, we also introduced the Scoped Storage feature to give users control over files and prevent apps from accessing sensitive user or app data. Your feedback has helped us refine this feature, and we recently announced several changes to make it easier to support. These are now available in Beta 3.

Another important change is restricting app launches from the background, which prevents apps from unexpectedly jumping into the foreground and taking over focus. In Beta 3 we’re transitioning from toast warnings to actually blocking these launches.

To prevent tracking we're limiting access to non-resettable device identifiers, including device IMEI, serial number, and similar identifiers. Read the best practices to choose the right identifiers for your use case. We're also randomizing MAC address when your device is connected to different Wi-Fi networks and gating connectivity APIs behind the location permission. We’re bringing these changes to you early, so you can have as much time as possible to prepare your apps.

Security

To keep users secure, we’ve extended our BiometricPrompt authentication framework to support biometrics at a system level. We're extending support for passive authentication methods such as face, and we’ve added implicit and explicit authentication flows. In the explicit flow, the user must explicitly confirm the transaction. The new implicit flow is designed for a lighter-weight alternative for transactions with passive authentication, and there’s no need for users to explicitly confirm.

Android Q also adds support for TLS 1.3, a major revision to the TLS standard that includes performance benefits and enhanced security. Our benchmarks indicate that secure connections can be established as much as 40% faster with TLS 1.3 compared to TLS 1.2. TLS 1.3 is enabled by default for all TLS connections made through Android’s TLS stack, called Conscrypt, regardless of target API level. See the docs for details.

Project Mainline

Today we also announced Project Mainline, a new approach to keeping Android users secure and their devices up-to-date with important code changes, direct from Google Play. With Project Mainline, we’re now able to update specific internal components within the OS itself, without requiring a full system update from your device manufacturer. This means we can help keep the OS code on devices fresher, drive a new level of consistency, and bring the latest AOSP code to users faster -- and for a longer period of time.

We plan to update Project Mainline modules in much the same way as app updates are delivered today -- downloading the latest versions from Google Play in the background and loading them the next time the phone starts up. The source code for the modules will continue to live in the Android Open Source Project, and updates will be fully open-sourced as they are released. Also, because they’re open source, they’ll include improvements and bug fixes contributed by our many partners and developer community worldwide.

For users, the benefits are huge, since their devices will always be running the latest versions of the modules, including the latest updates for security, privacy, and consistency. For device makers, carriers, and enterprises, the benefits are also huge, since they can optimize and secure key parts of the OS without the cost of a full system update.

For app and game developers, we expect Project Mainline to help drive consistency of platform implementation in key areas across devices, over time bringing greater uniformity that will reduce development and testing costs and help to make sure your apps work as expected. All devices running Android Q or later will be able to get Project Mainline, and we’re working closely with our partners to make sure their devices are ready.

Innovation and new experiences

Android is shaping the leading edge of innovation. With our ecosystem partners, we’re enabling new experiences through a combination of hardware and software advances.

Foldables

This year, display technology will take a big leap with foldable devices coming to the Android ecosystem from several top device makers. When folded these devices work like a phone, then you unfold a beautiful tablet-sized screen.

We’ve optimized Android Q to ensure that screen continuity is seamless in these transitions, and apps and games can pick up right where they left off. For multitasking, we’ve made some changes to onResume and onPause to support multi-resume and notify your app when it has focus. We've also changed how the resizeableActivity manifest attribute works, to help you manage how your app is displayed on large screens.

Our partners have already started showing their innovative foldable devices, with more to come. You can get started building and testing today with our foldables emulator in canary release of Android Studio 3.5.

5G networks

5G networks are the next evolution of wireless technology -- providing consistently faster speeds and lower latency. For developers, 5G can unlock new kinds of experiences in your apps and supercharge existing ones.

Android Q adds platform support for 5G and extends existing APIs to help you transform your apps for 5G. You can use connectivity APIs to detect if the device has a high bandwidth connection and check whether the connection is metered. With these your apps and games can tailor rich, immersive experiences to users over 5G.

With Android’s open ecosystem and range of partners, we expect the Android ecosystem to scale to support 5G quickly. This year, over a dozen Android device makers are launching 5G-ready devices, and more than 20 carriers will launch 5G networks around the world, with some already broad-scale.

Live Caption

On top of hardware innovation, we’re continuing to see Android’s AI transforming the OS itself to make it smarter and easier to use, for a wider range of people. A great example is Live Caption, a new feature in Android Q that automatically captions media playing on your phone.

Many people watch videos with captions on -- the captions help them keep up, even when on the go or in a crowded place. But for 466 million Deaf and Hard of Hearing people around the world, captions are more than a convenience -- they make content accessible. We worked with the Deaf community to develop Live Caption.

Live Caption brings real-time captions to media on your phone - videos, podcasts, and audio messages, across any app—even stuff you record yourself. Best of all, it doesn’t even require a network connection -- everything happens on the device, thanks to a breakthrough in speech recognition that we made earlier this year. The live speech models run right on the phone, and no audio stream ever leaves your device.

For developers, Live Caption expands the audience for your apps and games by making digital media more accessible with a single tap. Live Caption will be available later this year.

Suggested actions in notifications

In Android Pie we introduced smart replies for notifications that let users engage with your apps direct from notifications. We provided the APIs to attach replies and actions, but you needed to build those on your own.

Now in Android Q we want to make smart replies available to all apps right now, without you needing to do anything. Starting in Beta 3, we’re enabling system-provided smart replies and actions that are inserted directly into notifications by default.

You can still supply your own replies and actions if you want -- such as if you are using ML Kit or other ML frameworks. Just opt out of the system-provided replies or actions on a per-notification basis using setAllowGeneratedReplies() and setAllowSystemGeneratedContextualActions().

Android Q suggestions are powered by an on-device ML service built into the platform -- the same service that backs our text classifier entity recognition service. We’ve built it with user privacy in mind, and the ML processing happens completely on the device, not on a backend server.

Because suggested actions are based on the TextClassifier service, they can take advantage of new capabilities we’ve added in Android Q, such as language detection. You can also use TextClassifier APIs directly to generate system-provided notifications and actions, and you can mix those with your own replies and actions as needed.

Dark theme

Many users prefer apps that offer a UI with a dark theme they can switch to when light is low, to reduce eye strain and save battery. Users have also asked for a simple way to enable dark theme everywhere across their devices. Dark theme has been a popular request for a while, and in Android Q, it’s finally here.

Starting in Android Q Beta 3, users can activate a new system-wide dark theme by going to Settings > Display, using the new Quick Settings tile, or turning on Battery Saver. This changes the system UI to dark, and enables the dark theme of apps that support it. Apps can build their own dark themes, or they can opt-in to a new Force Dark feature that lets the OS create a dark version of their existing theme. All you have to do is opt-in by setting android:forceDarkAllowed="true" in your app’s current theme.

You may also want to take complete control over your app’s dark styling, which is why we’ve also been hard at work improving AppCompat’s DayNight feature. By using DayNight, apps can offer a dark theme to all of their users, regardless of what version of Android they’re using on their devices. For more information, see here.

Gestural navigation

Many of the latest Android devices feature beautiful edge-to-edge screens, and users want to take advantage of every bit of them. In Android Q we’re introducing a new fully gestural navigation mode that eliminates the navigation bar area and allows apps and games to use the full screen to deliver their content. It retains the familiar Back, Home, and recents navigation through edge swipes rather than visible buttons.

Users can switch to gestures in Settings > System > Gestures. There are currently two gestures: Swiping up from the bottom of the screen takes the user to the Home screen, holding brings up Recents. Swiping from the screen’s left or right edge triggers the Back action.

To blend seamlessly with gestural navigation, apps should go edge-to-edge, drawing behind the navigation bar to create an immersive experience. To implement this, apps should use the setSystemUiVisibility() API to be laid out fullscreen, and then handle WindowInsets as appropriate to ensure that important pieces of UI are not obscured. More information is here.

Digital wellbeing

Digital wellbeing is another theme of our work on Android -- we want to give users the visibility and tools to find balance with the way they use their phones. Last year we launched Digital Wellbeing with Dashboards, App Timers, Flip to Shush, and Wind Down mode. These tools are really helping. App timers helped users stick to their goals over 90% of the time, and users of Wind Down had a 27% drop in nightly usage.

This year we’re continuing to expand our features to help people find balance with digital devices, adding Focus Mode and Family Link.

Focus Mode

Focus Mode is designed for all those times you’re working or studying, and you want to to focus to get something done. With focus mode, you can pick the apps that you think might distract you and silence them - for example, pausing email and the News while leaving maps and text message apps active. You can then use Quick Tiles to turn on Focus Mode any time you want to focus. Under the covers, these apps will be paused - until you come out of Focus Mode! Focus Mode is coming to Android 9 Pie and Android Q devices this Fall.

Family Link

Family Link is a new set of controls to help parents. Starting in Android Q, Family Link will be built right into the Settings on the device. When you set up a new device for your child, Family Link will help you connect it to you. You’ll be able to set daily screen time limits, see the apps where your child is spending time, review any new apps your child wants to install, and even set a device bedtime so your child can disconnect and get to sleep. And now in Android Q you can also set time limits on specific apps… as well as give your kids Bonus Time if you want them to have just 5 more minutes at bedtime. Family Link is coming to Android P and Q devices this Fall. Make sure to check out the other great wellbeing apps in the recent Google Play awards.

Family link lets parents set device bedtime and even give bonus minutes.

Android Foundations

We’re continuing to extend the foundations of Android with more capabilities to help you build new experiences for your users -- here are just a few.

Improved peer-to-peer and internet connectivity

In Android Q we’ve refactored the Wi-Fi stack to improve privacy and performance, and also to improve common use-cases like managing IoT devices and suggesting internet connections -- without requiring the location permission. The network connection APIs make it easier to manage IoT devices over local Wi-Fi, for peer-to-peer functions like configuring, downloading, or printing. The network suggestion APIs let apps surface preferred Wi-Fi networks to the user for internet connectivity.

Wi-Fi performance modes

In Android Q apps can now request adaptive Wi-Fi by enabling high performance and low latency modes. These will be of great benefit where low latency is important to the user experience, such as real-time gaming, active voice calls, and similar use-cases. The platform works with the device firmware to meet the requirement with the lowest power consumption. To use the new performance modes, call WifiManager.WifiLock.createWifiLock().

Full support for Wi-Fi RTT accurate indoor positioning

In Android 9 Pie we introduced RTT APIs for indoor positioning to accurately measure distance to nearby Wi-Fi Access Points (APs) that support the IEEE 802.11mc protocol, based on measuring the round-trip time of Wi-Fi packets. Now in Android Q, we’ve completed our implementation of the 802.11mc standard, adding an API to obtain location information of each AP being ranged, configured by their owner during installation.

Audio playback capture

You saw how Live Caption can take audio from any app and instantly turn it into on-screen captions. It’s a seamless experience that shows how powerful it can be for one app to share its audio stream with another. In Android Q, any app that plays audio can let other apps capture its audio stream using a new API. In addition to enabling captioning and subtitles, the API lets you support popular use-cases like live-streaming games, all without latency impact on the source app or game.

We’ve designed this new capability with privacy and copyright protection in mind, so the ability for an app to capture another app's audio is constrained, giving apps full control over whether their audio streams can be captured. Read more here.

Dynamic depth for photos

Apps can now request a Dynamic Depth image which consists of a JPEG, XMP metadata related to depth related elements, and a depth and confidence map embedded in the same file on devices that advertise support. Requesting a JPEG + Dynamic Depth image makes it possible for you to offer specialized blurs and bokeh options in your app. You can even use the data to create 3D images or support AR photography use-cases. Dynamic Depth is an open format for the ecosystem -- the latest version of the spec is here. We're working with our device-maker partners to make it available across devices running Android Q and later.

With Dynamic Depth image you can offer specialized blurs and bokeh options in your app

New audio and video codecs

Android Q adds support for the open source video codec AV1, which allows media providers to stream high quality video content to Android devices using less bandwidth. In addition, Android Q supports audio encoding using Opus - a codec optimized for speech and music streaming, and HDR10+ for high dynamic range video on devices that support it. The MediaCodecInfo API introduces an easier way to determine the video rendering capabilities of an Android device. For any given codec, you can obtain a list of supported sizes and frame rates.

Vulkan 1.1 and ANGLE

We're continuing to expand the impact of Vulkan on Android, our implementation of the low-overhead, cross-platform API for high-performance 3D graphics. We’re working together with our device manufacturer partners to make Vulkan 1.1 a requirement on all 64-bit devices running Android Q and higher, and a recommendation for all 32-bit devices. For game and graphics developers using OpenGL, we’re also working towards a standard, updateable OpenGL driver for all devices built on Vulkan. In Android Q we're adding experimental support for ANGLE on top of Vulkan on Android devices. See the docs for details.

Neural Networks API 1.2

In NNAPI 1.2 we've added 60 new ops including ARGMAX, ARGMIN, quantized LSTM, alongside a range of performance optimisations. This lays the foundation for accelerating a much greater range of models -- such as those for object detection and image segmentation. We are working with hardware vendors and popular machine learning frameworks such as TensorFlow to optimize and roll out support for NNAPI 1.2.

Thermal API

When devices get too warm, they may throttle the CPU and/or GPU, and this can affect apps and games in unexpected ways. Now in Android Q, apps and games can use a thermal API to monitor changes on the device and take action to help restore normal temperature. For example, streaming apps can reduce resolution/bit rate or network traffic, a camera app could disable flash or intensive image enhancement, or a game could reduce frame rate or polygon tesselation. Read more here.

ART optimizations

Android Q introduces several improvements to the ART runtime to help your apps start faster, consume less memory, and run smoother -- without requiring any work from you. To help with initial app startup, Google Play is now delivering cloud-based profiles along with APKs. These are anonymized, aggregate ART profiles that let ART pre-compile parts of your app even before it's run. Cloud-based profiles benefit all apps and they're already available to devices running Android P and higher.

We’re also adding Generational Garbage Collection to ART's Concurrent Copying (CC) Garbage Collector. Generational CC collects young-generation objects separately, incurring much lower cost as compared to full-heap GC. It makes garbage collection more efficient in terms of time and CPU, reduces jank, and helps apps run better on lower-end devices.

More Android Q Beta devices, more Treble momentum than ever

In 2017 we launched Project Treble as part of Android Oreo, with a goal of accelerating OS updates. Treble provides a consistent, testable interface between Android and the underlying device code from device makers and silicon manufacturers, which makes porting a new OS version much simpler and more modular.

In 2018 we worked closely with our partners to bring the first OS updates to their Treble devices. The result: last year at Google I/O we had 8 devices from 7 partners joining our Android P Beta program, together with our Pixel and Pixel 2 devices. Fast forward to today -- we’re seeing updates to Android Pie accelerating strongly, with 2.5 times the footprint compared to Android Oreo's at the same time last year.

This year with Android Q we’re seeing even more momentum, and we have 21 devices from 12 top global partners joining us to release Android Q Beta 3 -- in addition all Pixel devices. We’re also providing Q Beta 3 Generic System Images (GSI), a testing environment for other supported Treble devices. All of these offer the same behaviors, APIs, and features -- giving you an incredible variety of devices for testing your apps, and more ways for you to get an early look at Android Q.

You can see the full list of supported partner and Pixel devices at android.com/beta. Try Android Q Beta on your favorite device today and let us know your feedback!

Explore the new features and APIs

When you're ready, dive into Android Q and learn about the new features and APIs you can use in your apps. Take a look at the API diff report for an overview of what's changed in Beta 3, and see the Android Q Beta API reference for details. Visit the Android Q Beta developer site for more resources, including release notes and how to report issues.

To build with Android Q, download the Android Q Beta SDK and tools into Android Studio 3.3 or higher, and follow these instructions to configure your environment. If you want the latest fixes for Android Q related changes, we recommend you use Android Studio 3.5 or higher.

How do I get Beta 3?

It's easy! Just enroll any Pixel device here to get the update over-the-air. If you're already enrolled, you'll receive the update soon, and, no action is needed on your part. Downloadable system images are also available.

You can also get Beta 3 on any of the other devices participating in the Android Q Beta program, from some of our top device maker partners. You can see the full list of supported partner and Pixel devices at android.com/beta. For each device you'll find specs and links to the manufacturer's dedicated site for downloads, support, and to report issues.

For even broader testing on supported devices, you can also get Android GSI images, and if you don’t have a device you can test on the Android Emulator -- just download the latest emulator system images via the SDK Manager in Android Studio.

As always, your input is critical, so please let us know what you think. You can use our hotlists for filing platform issues (including privacy and behavior changes), app compatibility issues, and third-party SDK issues. You've shared great feedback with us so far and we're working to integrate as much of it as possible in the next Beta release.

We're looking forward to seeing your apps on Android Q!

Android Q Scoped Storage: Best Practices and Updates

Posted by Jeff Sharkey, Software Engineer, and Seb Grubb, Product Manager

Application Sandboxing is a core part of Android’s design, isolating apps from each other. In Android Q, taking the same fundamental principle from Application Sandboxing, we introduced Scoped Storage.

Since the Beta 1 release, you’ve given us a lot of valuable feedback on these changes -- thank you for helping shape Android! Because of your feedback, we've evolved the feature during the course of Android Q Beta. In this post, we'll share options for declaring your app’s support for Scoped Storage on Android Q devices, and best practices for questions we've heard from the community.

Updates to help you adopt Scoped Storage

We expect that Scoped Storage should have minimal impact to apps following current storage best practices. However, we also heard from you that Scoped Storage can be an elaborate change for some apps and you could use more time to assess the impact. Being developers ourselves, we understand you may need some additional time to ensure your app’s compatibility with this change. We want to help.

In the upcoming Beta 3 release, apps that target Android 9 Pie (API level 28) or lower will see no change, by default, to how storage works from previous Android versions. As you update your existing app to work with Scoped Storage, you’ll be able to use a new manifest attribute to enable the new behavior for your app on Android Q devices, even if your app is targeting API level 28 or lower.

The implementation details of these changes will be available with the Beta 3 release, but we wanted to share this update with you early, so you can better prepare your app for Android Q devices. Scoped Storage will be required in next year’s major platform release for all apps, independent of target SDK level, so we recommend you add support to your app well in advance. Please continue letting us know your feedback and how we can better align Scoped Storage with your app’s use cases. You can give us input through this survey, or file bugs and feature requests here.

Best practices for common feedback areas

Your feedback is incredibly valuable and has helped us shape these design decisions. We also want to take a moment to share some best practices for common questions we’ve heard:

  • Storing shared media files. For apps that handle files that users expect to be sharable with other apps (such as photos) and be retained after the app has been uninstalled, use the MediaStore API. There are specific collections for common media files: Audio, Video, and Images. For other file types, you can store them in the new Downloads collection. To access files from the Downloads collection, apps must use the system picker.
  • Storing app-internal files. If your app is designed to handle files not meant to be shared with other apps, store them in your package-specific directories. This helps keep files organized and limit file clutter as the OS will manage cleanup when the app is uninstalled. Calls to Context.getExternalFilesDir() will continue to work.
  • Working with permissions and file ownership. For MediaStore, no permissions are necessary for apps that only access their own files. Your app will need to request permission to access media contributed by other apps. However, if your app is uninstalled and then reinstalled later, you’ll need to request permission from the user in order to be able to access media your app previously contributed.
  • Working with native code or libraries. The recommended pattern is to begin your media file discovery in your Java-based or Kotlin-based code, then pass the file's associated file descriptor into your native code.
  • Working with many files efficiently. If you need to perform bulk file operations in a single transaction, consider using ContentProvider.applyBatch(). Learn more about ContentProvider batch processing here.
  • Integrating with the system file picker.
    • Documents apps, such as a word processor, can use the ACTION_OPEN_DOCUMENT or ACTION_GET_CONTENT action to open a system file picker. You can learn more about the differences here.
    • File management apps typically work with collections of apps in a directory hierarchy. Use ACTION_OPEN_DOCUMENT_TREE to let the user pick a directory subtree. The app can further manipulate files available in the returned directory. Through this support, users can access files from any installed DocumentsProvider instance, which can be supported by any cloud-based or locally-backed storage solutions.

We’ve also provided a detailed Scoped Storage developer guide with additional information.

What’s ahead

It’s been amazing to see the community engagement on Android Q Beta so far. As we finalize the release in the next several months, please continue testing and keep the feedback coming. Join us at Google I/O 2019 for more details on Scoped Storage and other Android Q features. We’re giving a ”What’s new on Shared Storage” talk on May 8, and you’ll be able to find the livestream and recorded video on the Google I/O site.

Android Q Beta 2 update

Posted by Dave Burke, VP of Engineering

A few weeks ago we Introduced Android Q Beta, a first look at the next version of Android. Along with new privacy features for users, Android Q adds new capabilities for developers - like enhancements for foldables, new APIs for connectivity, new media codecs and camera capabilities, NNAPI extensions, Vulkan 1.1 graphics, and more.

Android's program of early, open previews is driven by our core philosophy of openness, and collaboration with our community. Your feedback since Beta 1 proves yet again the value of that openness - it's been loud, clear, and incredibly valuable. You've sent us thousands of bug reports, giving us insights and directional feedback, changing our plans in ways that make the platform better for users and developers. We're taking your feedback to heart, so please stay tuned. We're fortunate to have such a passionate community helping to guide Android Q toward the final product later this year.

Today we're releasing Android Q Beta 2 and an updated SDK for developers. It includes the latest bug fixes, optimizations, and API updates for Android Q, along with the April 2019 security patches. You'll also notice isolated storage becoming more prominent as we look for your wider testing and feedback to help us refine that feature.

We're still in early Beta with Android Q so expect rough edges! Before you install, check out the Known Issues. In particular, expect the usual transitional issues with apps that we typically see during early Betas as developers get their app updates ready. For example, you might see issues with apps that access photos, videos, media, or other files stored on your device, such as when browsing or sharing in social media apps.

You can get Beta 2 today by enrolling any Pixel device here. If you're already enrolled, watch for the Beta 2 update coming soon. Stay tuned for more at Google I/O in May.

What's new in Beta 2?

Privacy features for testing and feedback

As we shared at Beta 1, we're making significant privacy investments in Android Q in addition to the work we've done in previous releases. Our goals are improving transparency, giving users more control, and further securing personal data across platform and apps. We know that to reach those goals, we need to partner with you, our app developers. We realize that supporting these features is an investment for you too, so we'll do everything we can to minimize the impact on your apps.

For features like Scoped Storage, we're sharing our plans as early as possible to give you more time to test and give us your input. To generate broader feedback, we've also enabled Scoped Storage for new app installs in Beta 2, so you can more easily see what is affected.

With Scoped Storage, apps can use their private sandbox without permission, but they need new permissions to access shared collections for photos, videos and audio. Apps using files in shared collections -- for example, photo and video galleries and pickers, media browsing, and document storage -- may behave differently under Scoped Storage.

We recommend getting started with Scoped Storage soon -- the developer guide has details on how to handle key use-cases. For testing, make sure to enable Scoped Storage for your app using the adb command. If you discover that your app has a use-case that's not supported by Scoped Storage, please let us know by taking this short survey. We appreciate the great feedback you've given us already, it's a big help as we move forward with the development of this feature.

Bubbles: a new way to multitask

In Android Q we're adding platform support for bubbles, a new way for users to multitask and re-engage with your apps. Various apps have already built similar interactions from the ground up, and we're excited to bring the best from those into the platform, while helping to make interactions consistent, safeguard user privacy, reduce development time, and drive innovation.

Bubbles will let users multitask as they move between activities.

Bubbles help users prioritize information and take action deep within another app, while maintaining their current context. They also let users carry an app's functionality around with them as they move between activities on their device.

Bubbles are great for messaging because they let users keep important conversations within easy reach. They also provide a convenient view over ongoing tasks and updates, like phone calls or arrival times. They can provide quick access to portable UI like notes or translations, and can be visual reminders of tasks too.

We've built bubbles on top of Android's notification system to provide a familiar and easy to use API for developers. To send a bubble through a notification you need to add a BubbleMetadata by calling setBubbleMetadata. Within the metadata you can provide the Activity to display as content within the bubble, along with an icon (disabled in beta 2) and associated person.

We're just getting started with bubbles, but please give them a try and let us know what you think. You can find a sample implementation here.

Foldables emulator

As the ecosystem moves quickly toward foldable devices, new use-cases are opening up for your apps to take advantage of these new screens. With Beta 2, you can build for foldable devices through Android Q enhanced platform support, combined with a new foldable device emulator, available as an Android Virtual device in Android Studio 3.5 available in the canary release channel.

7.3" Foldable AVD switches between the folded and unfolded states

On the platform side, we've made a number of improvements in onResume and onPause to support multi-resume and notify your app when it has focus. We've also changed how the resizeableActivity manifest attribute works, to help you manage how your app is displayed on foldable and large screens. You can read more in the foldables developer guide.

To set up a runtime environment for your app, you can now configure a foldable emulator as a virtual device (AVD) in Android Studio. The foldable AVD is a reference device that lets you test with standard hardware configurations, behaviors, and states, as will be used by our device manufacturer partners. To ensure compatibility, the AVD meets CTS/GTS requirements and models CDD compliance. It supports runtime configuration change, multi-resume and the new resizeableActivity behaviors.

Use the canary release of Android Studio 3.5 to create a foldable virtual device to support either of two hardware configurations 7.3" (4.6" folded) and 8" (6.6" folded) with Beta 2. In each configuration, the emulator gives you on-screen controls to trigger fold/unfold, change orientation, and quick actions.

Android Studio - AVD Manager: Foldable Device Setup

Try your app on the foldable emulator today by downloading the canary release of Android Studio 3.5 and setting up a foldable AVD that uses the Android Q Beta 2 system image.

Improved sharesheet

Following on the initial Sharing Shortcuts APIs in Beta 1, you can now offer a preview of the content being shared by providing an EXTRA_TITLE extra in the Intent for the title, or by setting the Intent's ClipData for a thumbnail image. See the updated sample application for the implementation details.

Directional, zoomable microphones

Android Q Beta 2 gives apps more control over audio capture through a new MicrophoneDirection API. You can use the API to specify a preferred direction of the microphone when taking an audio recording. For example, when the user is taking a "selfie" video, you can request the front-facing microphone for audio recording (if it exists) by calling setMicrophoneDirection(MIC_DIRECTION_FRONT).

Additionally, this API introduces a standardized way of controlling zoomable microphones, allowing your app to have control over the recording field dimension using setMicrophoneFieldDimension(float).

Compatibility through public APIs

In Android Q we're continuing our long-term effort to move apps toward only using public APIs. We introduced most of the new restrictions in Beta 1, and we're making a few minor updates to those lists in Beta 2 to minimize impact on apps. Our goal is to provide public alternative APIs for valid use-cases before restricting access, so if an interface that you currently use in Android 9 Pie is now restricted, you should request a new public API for that interface.

Get started with Android Q Beta

Today's update includes Beta 2 system images for all Pixel devices and the Android Emulator, as well updated SDK and tools for developers. These give you everything you need to get started testing your apps on the new platform and build with the latest APIs.

First, make your app compatible and give your users a seamless transition to Android Q, including your users currently participating in the Android Beta program. To get started, just install your current app from Google Play onto a device or emulator running Beta 2 and work through the user flows. The app should run and look great, and handle the Android Q behavior changes for all apps properly. If you find issues, we recommend fixing them in the current app, without changing your targeting level. See the migration guide for steps and a recommended timeline.

With important privacy features that are likely to affect your apps, we recommend getting started with testing right away. In particular, you'll want to test against scoped storage, new location permissions, restrictions on background Activity starts, and restrictions on device identifiers. See the privacy checklist as a starting point.

Next, update your app's targetSdkVersion to 'Q' as soon as possible. This lets you test your app with all of the privacy and security features in Android Q, as well as any other behavior changes for apps targeting Q.

Explore the new features and APIs

When you're ready, dive into Android Q and learn about the new features and APIs you can use in your apps. Here's a video highlighting many of the changes for developers in Beta 1 and Beta 2. Take a look at the API diff report for an overview of what's changed in Beta 2, and see the Android Q Beta API reference for details. Visit the Android Q Beta developer site for more resources, including release notes and how to report issues.

To build with Android Q, download the Android Q Beta SDK and tools into Android Studio 3.3 or higher, and follow these instructions to configure your environment. If you want the latest fixes for Android Q related changes, we recommend you use Android Studio 3.5 or higher.

How do I get Beta 2?

It's easy - you can enroll here to get Android Q Beta updates over-the-air, on any Pixel device (and this year we're supporting all three generations of Pixel -- Pixel 3, Pixel 2, and even the original Pixel!). If you're already enrolled, you'll receive the update to Beta 2 soon, no action is needed on your part. Downloadable system images are also available. If you don't have a Pixel device, you can use the Android Emulator -- just download the latest emulator system images via the SDK Manager in Android Studio.

As always, your input is critical, so please let us know what you think. You can use our hotlists for filing platform issues (including privacy and behavior changes), app compatibility issues, and third-party SDK issues. You've shared great feedback with us so far and we're working to integrate as much of it as possible in the next Beta release.

What it looks like to process 3.5 million books in Google’s cloud

Today’s guest blog comes from Kalev Leetaru, founder of The GDELT Project, which monitors the
world’s news media in nearly every country in over 100 languages to identify the events and narratives driving our global society.

This past September I published into Google BigQuery a massive new public dataset of metadata from 3.5 million digitized English-language books dating back more than two centuries (1800-2015), along with the full text of 1 million of these books. The archive, which draws from the English-language public domain book collections of the Internet Archive and HathiTrust, includes full publication details for every book, along with a wide array of computed content-based data. The entire archive is available as two public BigQuery datasets, and there’s a growing collection of sample queries to help users get started with the collection. You can even map two centuries of books with a single line of SQL.

What did it look like to process 3.5 million books? Data-mining and creating a public archive of 3.5 million books is an example of an application perfectly suited to the cloud, in which a large amount of specialized processing power is needed for only a brief period of time. Here are the five main steps that I took to make the invaluable learnings of millions of books more easily and speedily accessible in the cloud:
  1. The project began with a single 8-core Google Compute Engine (GCE) instance with a 2TB SSD persistent disk that was used to download the 3.5 million books. I downloaded the books to the instance’s local disk, unzipped them, converted them into a standardized file format, and then uploaded them to Google Cloud Storage (GCS) in large batches, using the composite objects and parallel upload capability of GCS. Unlike traditional UNIX file systems, GCS performance does not degrade with large numbers of small files in a single directory, so I could upload all 3.5 million files into a common set of directories.
    Figure 1: Visualization of two centuries of books
  2. Once all books had been downloaded and stored into GCS, I launched ten 16-core High Mem (100GB RAM) GCE instances (160 cores total) to process the books, each with a 50GB persistent SSD root disk to achieve faster IO over traditional persistent disks. To launch all ten instances quickly, I launched the first instance and configured that with all of the necessary software libraries and tools, then created and used a disk snapshot to rapidly clone the other nine with just a few clicks. Each of the ten compute instances would download a batch of 100 books at a time to process from GCS.
  3. Once the books had been processed, I uploaded back into GCS all of the computed metadata. In this way, GCS served as a central storage fabric connecting the compute nodes. Remarkably, even in worst-case scenarios when all 160 processors were either downloading new batches of books from GCS or uploading output files back to GCS in parallel, there was no measurable performance degradation.
  4. With the books processed, I deleted the ten compute instances and launched a single 32-core instance with 200GB of RAM, a 10TB persistent SSD disk, and four 375GB direct-attached Local SSD Disks. I used this to reassemble the 3.5 million per-book output files into single output files, tab-delimited with data available for each year, merging in publication metadata and other information about each book. Disk IO of more than 750MB/s was observed on this machine.
  5. I then uploaded the final per-year output files to a public GCS directory with web downloading enabled, allowing the public to download the files.
Since very few researchers have the bandwidth, local storage or computing power to process even just the metadata of 3.5 million books, the entire collection was uploaded into Google BigQuery as a public dataset. Using standard SQL queries, you can explore the entire collection in tens of seconds at speeds of up to 45.5GB/s and perform complex analyses entirely in-database.

The entire project, from start to finish, took less than two weeks, a good portion of which consisted of human verification for issues with the publication metadata. This is significant because previous attempts to process even a subset of the collection on a modern HPC supercluster had taken over one month and completed only a fraction of the number of books examined here. The limiting factor was always the movement of data: transferring terabytes of books and their computed metadata across hundreds of processors.

This is where Google’s cloud offerings shine, seemingly purpose-built for data-first computing. In just two weeks, I was able to process 3.5 million books, spinning up a cluster of 160 cores and 1TB of RAM, followed by a single machine with 32 cores, 200GB of RAM, 10TB of SSD disk and 1TB of direct-attached scratch SSD disk. I was able to make the final results publicly accessible through BigQuery at query speeds of over 45.5GB/s.

You can access the entire collection today in BigQuery, explore sample queries, and read more technical detail about the processing pipeline on the GDELT Blog.

I’d like to thank Google, Clemson University, the Internet Archive, HathiTrust, and OCLC in making this project possible, along with all of the contributing libraries and digitization sponsors that have made these digitized books available.

- Posted by Kalev Leetaru, founder of The GDELT Project

Compute Engine now with 3 TB of high-speed Local SSD and 64 TB of Persistent Disk per VM

To help your business grow, we are significantly increasing size limits of all Google Compute Engine block storage products, including Local SSD and both types of Persistent Disk.

Now up to 64TB of Persistent Disk may be attached per VM for most machine types, including both Standard and SSD-backed Persistent Disk. The volume size limit has increased to 64 TB also, eliminating the need to stripe disks for larger volumes.

Persistent Disk provides fantastic price-performance and offers excellent usability for workloads that rely on durable block storage. Persistent Disk SSD delivers 30 IOPS per 1 GB provisioned, up to 15,000 IOPS per instance. Persistent Disk Standard is great value at $0.04 per GB-mo and provides 0.75 read IOPS per GB and 1.5 write IOPS per GB. Performance limits are set at an instance level, and can be achieved with just a single Persistent Disk.

We have also increased the amount of Local SSD that can be attached to a single virtual machine to 3 TB. Available in Beta today, you can attach twice as many partitions of Local SSD to Google Compute Engine instances. Up to eight 375 GB partitions or 3 TB of high IOPS SSD can now be attached to any machine with at least one virtual CPU.

We talked with Aaron Raddon, Founder and CTO at Lytics who tested our larger Local SSDs. He found they improved Cassandra performance by 50% and provide provisioning flexibility that can lead to additional savings.
The new, larger SSD has the same incredible IOPS performance we announced in January, topping out at 680,000 random 4K read IOPS and 360,000 random 4K write IOPS. With Local SSD you can achieve multiple millions of operations per second for key-value stores and a million writes per second using as few as 50 servers on NoSQL databases.

Local SSD retains the competitive pricing of $0.218 per GB/month while continuing to support extraordinary IOPS performance. As always, data stored in Local SSD is encrypted and our live migration technology means no downtime during maintenance. Local SSD also retains the flexibility of attaching to any instance type.

Siddharth Choudhuri, Principal Engineer at Levyx stated that doubling capacity on local SSDs with the same high IOPS is a game changer for businesses seeking low-latency and high throughput on large datasets. It enables them to index billions of objects on a single, denser node in real-time on Google Cloud Platform when paired with Levyx’s Helium data store.

To get started, head over to the Compute Engine console or read about Persistent Disk and Local SSD in the product documentation.

- Posted by John Barrus, Senior Product Manager, Google Cloud Platform