Cucumber tests parallel execution with Testlab

[anas-baadshah]

Problem:
I am running Cucumber UI tests with Firebase Testlab. All are working fine. The sequential run is successful and getting all reports.
Currently, I am trying to distribute the tests to run it on Testlab parallelly But seems like it is failing to find the tests.
I am getting some log which says 0 test cases found

0 test / 1 shard
1 matrix ids created in 0m 10s

It is actually run all the cases sequentially on the 1 shard created. But no parallel runs.
My Gcloud configuration looks like this

gcloud:
  # test and app are the only required args
  app: .apkpath
  test: .testapkPath
  auto-google-login: false
  use-orchestrator: true
  instrumentation: true
  environment-variables:
    coverage: true
    coverageFilePath: /sdcard/
    clearPackageData: true
  directories-to-pull:
    - /sdcard/
  performance-metrics: false
  num-flaky-test-attempts: 0
  device:
    - model: "NexusLowRes"
      version: 29

  results-bucket: bucket_name
  results-dir: results
  locale: en
  test-runner-class: CucumberInstrumentationRunner
  record-video: true
  timeout: 30m
  num-uniform-shards: 8 //Not working

Cucumber configuration:

@CucumberOptions(
        features = "features",
        glue = "com.test",
        tags = {"@mySevenTests"},
        monochrome = true)

class Hooks {
    @Rule
    var activityTestRule = ActivityTestRule(
            InitialActivity::class.java,
            shouldLaunchActivity = true,
            isInitialTouchMode = true)
            .with(MockWebServerExecutorTestTask())

    private var activity: Activity? = null

    @Before("@mySevenTests")
    fun setup(scenario: Scenario) {
            activityTestRule.launchActivity(null)
            activity = activityTestRule.activity
        }

    @After("@mySevenTests")
    fun tearDown(){
        activity?.finish()
    }
}

Cucumber Android version 4.8.4

Can someone please help on this.

[lsuski]

Check logcat logs from devices. You can paste it here

[anas-baadshah]

Check logcat logs from devices. You can paste it here

11-16 08:24:40.096: D/cucumber-android(11098): Found CucumberOptions in class com.test.CucumberInstrumentationRunner
11-16 08:24:41.412: D/TestExecutor(11098): Adding listener androidx.test.internal.runner.listener.LogRunListener
11-16 08:24:41.412: D/TestExecutor(11098): Adding listener androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener
11-16 08:24:41.412: D/TestExecutor(11098): Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
11-16 08:24:41.412: D/TestExecutor(11098): Adding listener androidx.test.internal.runner.listener.CoverageListener
11-16 08:24:41.413: I/TestRunner(11098): run started: 1 tests
11-16 08:24:41.516: I/TestRunner(11098): started: Feature name
11-16 08:24:41.583: D/cucumber-android(11098): Scenario Outline
11-16 08:24:41.586: D/cucumber-android(11098): Step 1
11-16 08:24:41.590: D/cucumber-android(11098): Step 2
11-16 08:24:42.103: D/cucumber-android(11098): Step 3
11-16 08:24:42.105: D/cucumber-android(11098): Step 4
11-16 08:24:42.108: D/cucumber-android(11098): Step 5
11-16 08:24:42.111: D/cucumber-android(11098): Step 6
......................................................
......................................................
......................................................
......................................................
11-16 08:24:42.167: D/cucumber-android(11098): Step 28
11-16 08:24:42.169: D/cucumber-android(11098): FINAL STEP
11-16 08:24:42.172: E/TestRunner(11098): failed: Scenerio Name
11-16 08:24:42.172: E/TestRunner(11098): ----- begin exception -----
11-16 08:24:42.172: E/TestRunner(11098): java.lang.RuntimeException: No activities found. Did you forget to launch the activity by calling getActivity() or startActivitySync or similar?
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.base.RootViewPicker.waitForAtLeastOneActivityToBeResumed(RootViewPicker.java:176)
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.base.RootViewPicker.get(RootViewPicker.java:88)
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.ViewInteractionModule.provideRootView(ViewInteractionModule.java:77)
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.ViewInteractionModule_ProvideRootViewFactory.provideRootView(ViewInteractionModule_ProvideRootViewFactory.java:37)
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.ViewInteractionModule_ProvideRootViewFactory.get(ViewInteractionModule_ProvideRootViewFactory.java:27)
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.ViewInteractionModule_ProvideRootViewFactory.get(ViewInteractionModule_ProvideRootViewFactory.java:10)
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.base.ViewFinderImpl.getView(ViewFinderImpl.java:63)
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.ViewInteraction.doPerform(ViewInteraction.java:223)
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.ViewInteraction.access$100(ViewInteraction.java:65)
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.ViewInteraction$1.call(ViewInteraction.java:158)
11-16 08:24:42.172: E/TestRunner(11098): 	at androidx.test.espresso.ViewInteraction$1.call(ViewInteraction.java:155)
11-16 08:24:42.172: E/TestRunner(11098): 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
11-16 08:24:42.172: E/TestRunner(11098): 	at android.os.Handler.handleCallback(Handler.java:883)
11-16 08:24:42.172: E/TestRunner(11098): 	at android.os.Handler.dispatchMessage(Handler.java:100)
11-16 08:24:42.172: E/TestRunner(11098): 	at android.os.Looper.loop(Looper.java:214)
11-16 08:24:42.172: E/TestRunner(11098): 	at android.app.ActivityThread.main(ActivityThread.java:7356)
11-16 08:24:42.172: E/TestRunner(11098): 	at java.lang.reflect.Method.invoke(Native Method)
11-16 08:24:42.172: E/TestRunner(11098): 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
11-16 08:24:42.172: E/TestRunner(11098): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:930)
11-16 08:24:42.172: E/TestRunner(11098): 	at ✽.STEP 2(file:features/myfeature/myseventests.feature:6)
11-16 08:24:42.172: E/TestRunner(11098): ----- end exception -----
11-16 08:24:42.175: I/TestRunner(11098): finished: Scenerio Name
11-16 08:24:42.248: E/cucumber-android(11098): java.lang.RuntimeException: No activities found. Did you forget to launch the activity by calling getActivity() or startActivitySync or similar?
11-16 08:24:42.248: I/TestRunner(11098): run finished: 1 tests, 1 failed, 0 ignored

Even though the last log entry says test failed, Actually the test succeeded and marked it as passed on Testlab. 🧐🧐

[lsuski]

Exception suggests that no activity is started. Maybe Hooks.setup is not called? Try to put some logcat logs in your methods to see what is exactly going on

[anas-baadshah]

Exception suggests that no activity is started. Maybe Hooks. setup is not called? Try to put some logcat logs in your methods to see what is exactly going on

I put logs on both @before and @after invocations and in between each test run, It is printing the logs. Also, The execution was successful even on Testlab and local. The only issue is that the tests are not distributing to run on their own shards on Testlab. It always executes tests sequentially. I am trying to split and run the cucumber tests on each device on Firebase Testlab.

Log @after and @before

12-16 08:24:41.516: HOOKS_LOG: Before invoked
12-16 08:24:41.583: HOOKS_LOG: After invoked
12-16 08:24:41.586: HOOKS_LOG: Before invoked
12-16 08:24:41.590: HOOKS_LOG: After invoked
12-16 08:24:42.103: HOOKS_LOG: Before invoked
12-16 08:24:42.105: HOOKS_LOG: After invoked
12-16 08:24:42.108: HOOKS_LOG: Before invoked
12-16 08:24:42.111: HOOKS_LOG: After invoked

[lsuski]

I am not familiar with Firebase Test Lab. If you run tests locally with simple instrumentation commandline and specified shard arguments then it should work (or with spoon runner)

[anas-baadshah]

I am not familiar with Firebase Test Lab. If you run tests locally with simple instrumentation Commandline and specified shard arguments then it should work (or with spoon runner)

Thanks @lsuski. I can try that. Do we have documentation to try this? This way even if we use any CI system to run our cucumber tests, It will consume the local emulators to share the tests if I understood correctly.

[lsuski]

Check android documentation for running shards or spoon runner on github or just cukeulator sample in this repo.