Exception before running instrumentation tests on Android 19 and below

[aaalaniz]

Description

I am seeing an exception when using the JunitParams runner on Android 19 and below.

Steps to Reproduce

1. Write simple test suite and execute on Android 19 or lower with latest Android Testing Support Libaries

Code

The following test suite will reproduce the issue

@RunWith(JUnitParamsRunner.class)
public class SimpleTest {
    @Test
    @Parameters({ "false", "true" })
    public void simpleParameterizedTest(boolean enabled) {
        // Should execute and pass
    }
}

Expected Behavior

Tests should execute and pass. (Android Studio screenshot below)

Actual Behavior

Tests are not run because "no tests are found" (Android Studio screenshot below)

Logs

I checked the logs and I see the following exception occurring.

09-07 12:17:01.187 3338-3355/com.twilio.video.test W/dalvikvm: Exception Ljava/util/regex/PatternSyntaxException; thrown while initializing Ljunitparams/naming/MacroSubstitutionNamingStrategy;
09-07 12:17:01.187 3338-3355/com.twilio.video.test E/TestExecutor: Fatal exception when running tests
                                                                   java.lang.ExceptionInInitializerError
                                                                       at junitparams.internal.TestMethod$1.computeValue(TestMethod.java:40)
                                                                       at junitparams.internal.TestMethod$1.computeValue(TestMethod.java:35)
                                                                       at junitparams.internal.Memoizer.get(Memoizer.java:11)
                                                                       at junitparams.internal.TestMethod.describe(TestMethod.java:116)
                                                                       at junitparams.internal.ParameterisedTestClassRunner.describeParameterisedMethod(ParameterisedTestClassRunner.java:161)
                                                                       at junitparams.JUnitParamsRunner.describeMethod(JUnitParamsRunner.java:502)
                                                                       at junitparams.internal.ParametrizedTestMethodsFilter.filteredMethods(ParametrizedTestMethodsFilter.java:30)
                                                                       at junitparams.JUnitParamsRunner.getListOfMethods(JUnitParamsRunner.java:498)
                                                                       at junitparams.JUnitParamsRunner.getDescription(JUnitParamsRunner.java:487)
                                                                       at org.junit.runners.Suite.describeChild(Suite.java:123)
                                                                       at org.junit.runners.Suite.describeChild(Suite.java:27)
                                                                       at org.junit.runners.ParentRunner.shouldRun(ParentRunner.java:434)
                                                                       at org.junit.runners.ParentRunner.filter(ParentRunner.java:382)
                                                                       at org.junit.runner.manipulation.Filter.apply(Filter.java:97)
                                                                       at android.support.test.internal.runner.TestRequestBuilder$LenientFilterRequest.getRunner(TestRequestBuilder.java:413)
                                                                       at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
                                                                       at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58)
                                                                       at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:375)
                                                                       at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
                                                                    Caused by: java.util.regex.PatternSyntaxException: Look-behind pattern matches must have a bounded maximum length near index 40:
                                                                   (?=\{[^\}]{0,50}\})|(?<=\{[^\}]{0,50}\})
                                                                                                           ^
                                                                       at java.util.regex.Pattern.compileImpl(Native Method)
                                                                       at java.util.regex.Pattern.compile(Pattern.java:411)
                                                                       at java.util.regex.Pattern.<init>(Pattern.java:394)
                                                                       at java.util.regex.Pattern.compile(Pattern.java:381)
                                                                       at junitparams.naming.MacroSubstitutionNamingStrategy.<clinit>(MacroSubstitutionNamingStrategy.java:13)
                                                                       	... 19 more

Versions

Here is a snippet from my build.gradle file.

androidTestCompile 'pl.pragmatists:JUnitParams:1.1.0'
androidTestCompile 'com.android.support.test.espresso:espresso-core:3.0.1'
androidTestCompile 'com.android.support.test:runner:1.0.1'
androidTestCompile 'com.android.support.test:rules:1.0.1'
androidTestCompile 'com.android.support.test.uiautomator:uiautomator-v18:2.1.2'

[mattmook]

I've created a PR #139 to fix this issue by replacing the offending RegEx with a simple state machine. The issue occurs because older versions of Android don't support look-behind properly.

[marmatys]

@aaalaniz thanks for reporting issue.
I created a repo https://github.com/marmatys/JUnitParamsRegexpAndroid for reproducing bug reported by you. It contains simple Android project.

@mattmook thank you for your PR. I merged you PR to my local branch, build JUnitParams from this version and then I use this snapshot version in repository mentioned earlier. Unfortunately when I now run android instrumentation tests I get following error:

TestExecutor: Fatal exception when running tests
TestExecutor: java.lang.ArrayIndexOutOfBoundsException: length=1; index=-1
TestExecutor: 	at java.util.ArrayList.get(ArrayList.java:439)
TestExecutor: 	at junitparams.internal.ParameterisedTestMethodRunner.currentTestDescription(ParameterisedTestMethodRunner.java:110)
TestExecutor: 	at junitparams.internal.ParameterisedTestClassRunner.getDescriptionFor(ParameterisedTestClassRunner.java:189)
TestExecutor: 	at junitparams.JUnitParamsRunner.describeChild(JUnitParamsRunner.java:455)
TestExecutor: 	at junitparams.JUnitParamsRunner.describeChild(JUnitParamsRunner.java:393)
TestExecutor: 	at org.junit.runners.ParentRunner.shouldRun(ParentRunner.java:434)
TestExecutor: 	at org.junit.runners.ParentRunner.filter(ParentRunner.java:382)
TestExecutor: 	at junitparams.JUnitParamsRunner.filter(JUnitParamsRunner.java:406)
TestExecutor: 	at org.junit.runner.manipulation.Filter.apply(Filter.java:97)
TestExecutor: 	at org.junit.runners.ParentRunner.filter(ParentRunner.java:384)
TestExecutor: 	at org.junit.runner.manipulation.Filter.apply(Filter.java:97)
TestExecutor: 	at android.support.test.internal.runner.TestRequestBuilder$LenientFilterRequest.getRunner(TestRequestBuilder.java:413)
TestExecutor: 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
TestExecutor: 	at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58)
TestExecutor: 	at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:375)
TestExecutor: 	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2074)

This problem even occurs on Android version bigger than 19. So problem needs to be investigated further. If I miss something please let me know. Any help is appreciated.

[mattmook]

The bug with the regexp existed in v1.0.6 however it seemed v1.1.0 introduced this ArrayIndexOutOfBoundsException regardless of the regexp fix. As such, I was considering it a separate issue as it exists independently.

I don't know enough about the internals to know what ParameterisedTestMethodRunner does with currentTestDescription, however locally I had patched it with the following:

Description currentTestDescription() {
    try {
        return method.description().getChildren().get(count - 1);
    } catch (IndexOutOfBoundsException ex) {
        return null;
    }
}

Clearly, this just masks the issue rather than resolves it. As count starts at 0 it is implying that currentParamsFromAnnotation has not been called to ensure it is incremented to 1 before currentTestDescription is called.