By Simran Basi, Dan Shi, Dan Willemsen, and Clay Murphy

Android Engineering Productivity (Android EngProd) seeks to ease development of the Android operating system for the entire ecosystem. Android EngProd creates tools, processes, and documentation aimed at Android platform development. We are now starting to push the best previously internal development infrastructure into the open for all to benefit.

Although comprehensive, the Android Compatibility Test Suite (CTS) and Trade Federation Test Harness can be unwieldy to configure. So we recently publicly released new tooling and associated docs that simplify device configuration and testing in the form of the Soong build system replacing Make, Test Mapping for easy configs, and Atest to run tests locally.

Configuring tests in Soong builds

The Soong build system was introduced in Android 8.0 (Oreo) to eventually replace the Make-based system (i.e. Android.mk files) used in previous releases. Soong allows simple build configuration with support for android_test declarations arriving in Android Q, now available in the Android Open Source Project (AOSP) master branch.

Soong uses Android.bp files, which are JSON-like simple declarative descriptions of modules to build. Here is an example test configuration in Soong, from: /platform_testing/tests/example/instrumentation/Android.bp
Note the android_test declaration at the beginning indicates this is a test. Including android_app instead would conversely indicate this is a build package. Complex test configuration options still exist for test modules requiring customized setup and tear down that cannot be performed within the test case itself.

Mapping tests in the source tree

Test Mapping allows developers to create pre- and post-submit test rules directly in the Android source tree and leave the decisions of branches and devices to be tested to the test infrastructure itself. Test Mapping definitions are JSON files named TEST_MAPPING that can be placed in any source directory.

Test Mapping categorizes tests via a test group. The name of a test group can be any string. For example, presubmit can be for a group of tests to run when validating changes. And postsubmit tests can be used to validate the builds after changes are merged.

For the directory requiring test coverage, simply add a TEST_MAPPING JSON file resembling the example below. These rules will ensure the tests run in presubmit checks when any files are touched in that directory or any of its subdirectories.

Here is a sample TEST_MAPPING file:
{
  "presubmit": [
    {
      "name": "CtsAccessibilityServiceTestCases",
      "options": [
        {
          "include-annotation": "android.platform.test.annotations.Presubmit"
        }
      ]
    }
  ],
  "postsubmit": [
    {
      "name": "CtsWindowManagerDeviceTestCases"
    }
  ],
  "imports": [
    {
      "path": "frameworks/base/services/core/java/com/android/server/am"
    }
  ]
}

Running tests locally with Atest

Atest is a command line tool that allows developers to build, install, and run Android tests locally, greatly speeding test re-runs without requiring knowledge of Trade Federation Test Harness command line options.
Atest commands take the following form:
atest [optional-arguments] test-to-run
You can run one or more tests by separating test references with spaces, like so:
atest test-to-run-1 test-to-run-2

To run an entire test module, use its module name. Input the name as it appears in the LOCAL_MODULE or LOCAL_PACKAGE_NAME variables in that test's Android.mk or Android.bp file.
atest FrameworksServicesTests
atest CtsJankDeviceTestCases

Discovering tests with Atest and TEST MAPPING

Atest and TEST MAPPING work together to solve the problem of test discovery, i.e. what tests need to be run when a directory of code is edited. For example, to execute all presubmit test rules for a given directory locally:

1. Go to the directory containing the TEST_MAPPING file.
2. Run the command: atest

Finding more testing documentation

Further, introductory testing documents were published on source.android.com to support Soong and platform testing in general:

• Android Platform Testing
• Test Development Workflow
• Simple Test Configuration
• Instrumentation Tests
• Native Tests
• JAR (Java) Host Tests

In addition to exposing more testing documentation, Android has recently opened up build infrastructure to monitor submissions through ci.android.com. See the More visibility into the Android Open Source Project blog post and the Continuous Integration Dashboard for instructions on viewing build status and downloading build artifacts.
Android EngProd endeavors to bring you more previously internal-only features to make your life easier. Watch this Google Testing Blog, the Android Developers Blog, and source.android.com for future enhancements.