Android Test Orchestrator - cucumber test report is deleted after each test

[pawelgiczewski]

Steps:

1. Setup Android project with cucumber android - create several tests
2. Enable Android Test Orchestrator + set clear package data after each test to true (I followed setup from cukeulator)
3. Run tests using adb and android test orchestrator (hint: https://developer.android.com/training/testing/junit-runner#ato-command-line)
4. Pull test reports from device

Result:
4. Report contains only the last test

Expected:
4. Report should contain all tests

The reason seems to be quite straightforward - package data is deleted after each test by test orchestrator. I tried to setup another locations for the reports, but I can only write to external storage belonging to the package. Any idea how to workaround that?

[lsuski]

Did you try adding pwrmission to write on external storage, granting it before each test?

[lsuski]

By reports you mean custom cucumber reports configured in @CucumberOptions? What about default gradle report generated by connected...Task?

[pawelgiczewski]

Did you try adding pwrmission to write on external storage, granting it before each test?

Generally, following the example from cukeulator. We don't need those permissions. Take a look at CukeulatorAndroidJunitRunner code:
//sdcard/Android/data/cucumber.cukeulator File directory = getTargetContext().getExternalFilesDir(null);
App does not need any special permissions to access this directory as it belongs to the package itself. However, if we set clearPackageData to true, this directory is cleared between tests by adb shell pm clear com.mypackage

I tried to modify the report path as follows:
File directory = Environment.getExternalStorageDirectory();
and in this case I added permissions for both packages (regular and test package) in App Manifest. I also accepted those permissions via adb directly before running the tests:
adb -s $EMU_ID shell pm grant com.mypackage.test android.permission.WRITE_EXTERNAL_STORAGE
adb -s $EMU_ID shell pm grant com.mypackage android.permission.WRITE_EXTERNAL_STORAGE

Without success. I cannot access those directories.

By reports you mean custom cucumber reports configured in @CucumberOptions? What about default gradle report generated by connected...Task?

Yes, I mean custom report. Default gradle report is generated correctly. However, by design it is missing some useful information which is delivered within cucumber report.

---

The only workaround I found so far, is to set reports path File directory = targetContext.getObbDir() (this directory is not cleared by clear command). Obb stands for some extension files added to huge APKs.

This leads to the next issue. Even if reports are stored in obb dir, they seem to be cleared by cucumber itself on each test startup (maybe it is caused by the way in which orchestrator starts cucumber tests?). I investigated that by adding timestamp to report directories. This way we can preserve several reports after one run and then could merge them - really ugly solution :)

[lsuski]

To be honest I don't use cucumber reports and orchestrator in my projects so I can't help much in this case. I would check how android gradle plugin generates its reports that they are not deleted after each test

[pawelgiczewski]

Got it. To be precise - gradle plugin does not generate the report. Report is generated by the runner (in this case Android Test Orchestrator). As I briefly analysed sources of Android Test Orchestrator, I can see that it generates reports on sdcard/odo path (not related to any package so it is not cleared) (https://github.com/android/android-test/blob/master/runner/android_test_orchestrator/java/androidx/test/orchestrator/listeners/OrchestrationXmlTestRunListener.java#L106 It uses methods I tried in above comments without success (however, it somehow grants writing permissions in runtime).

I'll try to investigate issue a bit deeper. It would be awesome to get it working.

[mkonkel]

@pawelgiczewski any progress?

[pawelgiczewski]

Nope. Finally, I did not find any solution for that.

For now, I stayed with XML reports generated by Android Junit Runner.

[childnode]

honestly, this is one of the most annoying, showstopping issues in '22

to differentiate between the issues:

• Test results are possibly deleted when usingandroid { defaultConfig { testInstrumentationRunnerArguments clearPackageData: 'true' } } will clear after (complete) test execution
  • you can mitigate this by setting this false
  • you can mitigate this by using additional_test_output dir in media https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:build-system/gradle-core/src/main/java/com/android/build/gradle/internal/testing/utp/AdditionalTestOutputUtils.kt;l=53?q=additional_test_output
  • latter one is needed anyway on > R to mitigate the scoped storage
  • combined use @CucumberOptions( plugin = [ "pretty", "json:/sdcard/Android/media/com.example.demo/additional_test_output/cucumber.json", "junit:/sdcard/Android/media/com.example.demo/additional_test_output/cucumber.xml" ], )
    • this will auto-pull additional output from device to your projects build/outputs - WARNING do not use any subfolders, because AGP (ADB PULL) will fail with Failed to retrieve additional test outputs from device. com.android.ddmlib.SyncException: read failed: Is a directory - i.e. do not use html reports - see https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-main:build-system/gradle-core/src/main/java/com/android/build/gradle/internal/testing/SimpleTestRunnable.java;l=245?q=SimpleTestRunnable
• when switching from android { testOptions { execution 'host' } } (default) to android { testOptions { execution 'android_test_orchestrator' } } or androidx_test_orchestrator the "each test overrides the result of any previous test" happens! (see https://developer.android.com/reference/tools/gradle-api/7.1/com/android/build/api/dsl/TestOptions#execution ) ... this is "by design" to me as long you use the same file for any instrumentation! see https://developer.android.com/training/testing/instrumented-tests/androidx-test-libraries/runner#use-android
  • at least, instrumentation with orchestrator seems to skip the clearPackageData: 'false' or somehow behaves same same but different!! ... honestly ... I don't get it yet at this point ... looking forward to debug more into
  • see custom InstrumentationRunner below using dynamic report paths in additional_test_outputs as it works to me actually
• https://github.com/cucumber/cucumber-android/blame/81d4b711d3c3d6e3623a59bd1204413b7c505487/README.md#L41 CucumberOptions.format is wrong while cucumber-java 4.8.1

  cucumber-android/build.gradle

  Line 49 in 6838037

  ext.cucumber_javaVersion = '4.8.1'

  uses plugin https://github.com/cucumber/cucumber-jvm/blob/v4.8.1/junit/src/main/java/io/cucumber/junit/CucumberOptions.java#L82

android {
    defaultConfig {
        applicationId "com.example.demo"

        testApplicationId "com.example.demo.uitest"

        // testInstrumentationRunner "io.cucumber.android.runner.CucumberAndroidJUnitRunner"
        testInstrumentationRunner "com.example.demo.uitest.InstrumentationRunner"

        testInstrumentationRunnerArguments clearPackageData: 'false'
    }
    adbOptions {
        installOptions '-g', '-r'
    }
    testOptions {
        execution 'androidx_test_orchestrator'
    }
}
dependencies {
    androidTestImplementation "io.cucumber:cucumber-android:4.9.0"
}

package com.example.demo.uitest

import android.os.Build
import android.os.Bundle
import io.cucumber.android.runner.CucumberAndroidJUnitRunner
import io.cucumber.junit.CucumberOptions
import java.io.File
import java.util.*

@CucumberOptions(
    features = ["features"],
    glue = ["com.example.demo.cucumber.steps"],
)
class InstrumentationRunner : CucumberAndroidJUnitRunner() {

    init {
        INSTANCE = this
    }

    /**
     * if additionalTestOutputDir is defined for DeviceProviderInstrumentTestTask this is
     * set as testInstrumentationRunnerArgument from SimpleTestRunnable
     */
    private var skipReportSubpathForResultOnSimpleTestRunnable : Boolean = false

    override fun onCreate(bundle: Bundle) {
        skipReportSubpathForResultOnSimpleTestRunnable = bundle.getString("additionalTestOutputDir").let { it != null && it.isNotEmpty() }
        bundle.putString(
            "plugin",
            pluginConfiguration
        )
        super.onCreate(bundle)
    }

    /**
     * Prepares the configuration string for saving the result. It will save the results in three formats:
     * xmk, html and json.
     * All results will be saved on device. See the path building method: [.getAbsoluteFilesPath]
     *
     * @return @[String] with plugin configuration
     */
    private val pluginConfiguration: String
        get() {
            val filename = "cucumber"
            val seperator = "--"
            return (
                "junit:$reportsPath/$filename.${UUID.randomUUID()}.xml" +
                    seperator +
                // TODO: disabled due to SimpleTestRunnable can't copy paths
                // "html:$reportsPath/cucumber-reports/html-report" + seperator +
                "json:$reportsPath/$filename.${UUID.randomUUID()}.json"
                )
        }

    companion object {

        private lateinit var INSTANCE: InstrumentationRunner

        /**
         * Builds the path in which the result reports will be saved
         * matches default location from {@link com.android.builder.internal.testing.SimpleTestRunnable#queryAdditionalTestOutputLocation()} line #347
         *
         * TODO: this changes on API 30 https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-master-dev:build-system/gradle-core/src/main/java/com/android/build/gradle/internal/testing/SimpleTestRunnable.java;l=326?q=%22fetching%20coverage%20dir%20from%22
         * @return @[String] path: <device>/<packagename>/files/test_data
         */
        val testDataPath: File
            get() =
                if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.R) {
                    // see https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-master-dev:build-system/gradle-core/src/main/java/com/android/build/gradle/internal/testing/SimpleTestRunnable.java;l=326?q=%22fetching%20coverage%20dir%20from%22
                    // TODO android.provider.MediaStore.Files.getContentUri(android.provider.MediaStore.VOLUME_EXTERNAL).toFile()
                    //  ?!
                    File(
                        "/sdcard/Android/media/" + INSTANCE.targetContext.applicationInfo.packageName + "/additional_test_output"
                    )
                } else {
                    // see https://cs.android.com/android-studio/platform/tools/base/+/mirror-goog-studio-master-dev:build-system/gradle-core/src/main/java/com/android/build/gradle/internal/testing/SimpleTestRunnable.java;l=364?q=%22fetching%20coverage%20dir%20from%22
                    File(
                        INSTANCE.targetContext.getExternalFilesDir(null),
                        "test_data"
                    ).apply { mkdirs() }
                }

        /**
         * Builds the path in which the result reports will be saved
         *
         * @return @[String] path: ${testDataPath}/reports or ${testDataPath}
         */
        val reportsPath: File
            get() = if (INSTANCE.skipReportSubpathForResultOnSimpleTestRunnable) testDataPath
            else File(
                testDataPath,
                "reports"
            ).apply { mkdirs() }
    }
}

[ThomasRichtsfeld]

Any updates on that from Cucumber side?
Nowadays most folks use TestOrchestrator to run their UI tests, so this should have a high priority

[lsuski]

Not on my side, I don't use orchestrator because it's slow. What I see is that the only way to preserve reports is to save them outside app package (?) which requires external storage write permission?

[ThomasRichtsfeld]

@pawelgiczewski how did you merge the different test results? We want to upload the test report as json file to AssertThat. Unfortunately, we end up with x different files depending on the scenarios. The same thing you mentioned in one of your comments.
I really wonder why there is no solution to that problem for 3 years. The test orchestrator is the recommended way from Google when executing UI tests. This is especially important if you have login scenarios that need to be reset after each test run

[kaoru2010]

As a workaround, I was able to merge the report files (in JSON format) into a single file using multiple-cucumber-html-reporter.

I've created a sample project that combines Cucumber for Android, Android Test Orchestrator, and multiple-cucumber-html-reporter. (tested on a MacBook Pro (Apple chip) and an Android 14 (API 34) emulator.)

https://github.com/kaoru2010/cucumber-android_android-test-orchestrator_demo

As an additional note, this sample uses TestStorage, so no extra permissions are required.

[johngray1965]

I don't believe this is a cucumber bug.

if you set useTestStorageService to true, and include test services dependency:

androidTestUtil("androidx.test.services:test-services:1.4.2")

and set:
enableAndroidTestCoverage = true

Then I get the files to the local host.