Roadmap/plan for refactoring tests

[andyscott]

I'd like to begin work refactoring the tests. This ticket serves as a place to begin discussion on how to go about this work.

At the very least, I'd love to see the tests split up so that we can run them individually. This will ease development on the rules.

Additionally, I'd love to have a way to run the same tests while varying configuration-- mostly this means varying toolchains and/or specific providers passed to targets while running whole suites of tests. Protobuf fails on windows, currently, so imagine being able to skip the protobuf suite for windows.

Possible approaches:

• bash scripts
  This would work, but runs the risk of being overly complicated and/or hard to maintain. It was somewhat successful in https://github.com/higherkindness/rules_scala. Additionally, these tests sit outside Bazel and there's no caching for unhappy path tests.

• https://github.com/bazelbuild/bazel-integration-testing.
  I don't have much info here, but my understanding is that @ittaiz is quite experienced with it. I have a personal TODO to investigate this further.

[ittaiz]

Indeed. The infra cost is not small unfortunately but we’ve gained a lot from using this library. Remember that current tests rely to some degree on the fact that they’re not isolated.

On Mon, 19 Aug 2019 at 22:19 Andy Scott ***@***.***> wrote: I'd like to begin work refactoring the tests. This ticket serves as a place to begin discussion on how to go about this work. At the very least, I'd love to see the tests split up so that we can run them individually. This will ease development on the rules. - @ittaiz <https://github.com/ittaiz> has mentioned using https://github.com/bazelbuild/bazel-integration-testing — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#824?email_source=notifications&email_token=AAKQQF5WMK3UNDAR6KCLEWDQFLW2DA5CNFSM4INE526KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HGCDD5A>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKQQF4NCUSBPBZ4NCUG4NDQFLW2DANCNFSM4INE526A> .

-- *Ittai Zeidman* Cell: 054-6735021 40 Hanamal street, Tel Aviv, Israel <http://www.wix.com>

[borkaehw]

@andyscott Hi Andy, I am also interested in this issue. Have you had a better idea/plan by any chance?

[borkaehw]

I have done some experiments on both methods proposed by @andyscott . Both seem like good candidates, but I am leaning towards the bash script approach. While the caching concerns are valid, we can get around this by wrapping the scripts with sh_test. This is what bazelbuild/bazel does for their integration tests.

I'm concerned that the learning curve for bazel-integration-testing is a bit high. I ran into some issues trying to set up a minimal example. I'm also concerned that the migration effort would likely be larger than splitting out the tests into separate bash scripts and wrapping those with sh_test.

As for varying toolchains and/or specific providers, I am not quite sure which approach is better.

[SrodriguezO]

I think this is a great idea. A good first step might be to go through test_rules_scala.sh and group tests/logic that belong together and/or depend on each other. Then we can split these groups into their own scripts with a corresponding sh_test rule in the BUILD file.

@andyscott @ittaiz wdyt?
@borkaehw and I can start on this next week if it sounds like a good approach.

[ittaiz]

I think Bazel's sh tests are legacy. They're moving <http://julio.meroh.net/2018/03/bazel-tests-in-java-part-1.html> to java and when I discussed the sh framework with them a few years ago they suggested I start the bazel-integration-testing project. Additionally many of our issues weren't just with the file being long but from bash itself. It's just not a maintainable language. I think we should put our efforts into moving away from bash to something else. If we'll agree on python or go I can be aboard (though I think it would be a mistake for a scala/jvm based audience). Lastly if there are specific AIs to help lower the entry barrier for the bazel-integration-testing library I'd be more than happy to do that.

…

On Tue, Sep 17, 2019 at 3:09 AM Sergio Rodriguez Orellana < ***@***.***> wrote: I think this is a great idea. A good first step might be to go through test_rules_scala.sh and group tests/logic that belong together and/or depend on each other. Then we can split these groups into their own scripts with a corresponding sh_test rule in the BUILD file. @andyscott <https://github.com/andyscott> @ittaiz <https://github.com/ittaiz> wdyt? Kevin and I can start on this next week if it sounds like a good approach. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#824?email_source=notifications&email_token=AAKQQF4RPT7ZGVYGQ5QRS7DQKAN2HA5CNFSM4INE526KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6237PA#issuecomment-532004796>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKQQF6RA5L5ZME25EXPMI3QKAN2HANCNFSM4INE526A> .

[SrodriguezO]

Ah. Python's probably not a great idea either, since we'd still have to transcribe all the tests and in the end we'd land in a non-JVM language.. In that case it might be worth revisiting bazel-integration-testing, since we'd be able to keep the tests in a JVM language that way. (Quick aside though, we ran into some early issues: bazelbuild/bazel-integration-testing#153)

As an intermediate step, however, I still think it would be worthwhile to break up test_rules_scala.sh. We'll need to do this regardless, and migrating individual tests will be less daunting than trying to do them all at once. I suggest we break it up into small shell scripts as the first step and then migrate those to whichever framework we decide on.

This will also give us the benefit of being able to run individual tests sooner, while we migrate to something like bazel-integration-testing.

Does this sound reasonable @ittaiz ?

[ittaiz]

It might but I think the cost of breaking up the bash while verifying it still works AND making the tests standalone (they’re not right now) is way too big then you think. I think it’s not the right move but as intermediate one I’m not completely against it. I’ll take a look soon at 153

On Wed, 18 Sep 2019 at 2:05 Sergio Rodriguez Orellana < ***@***.***> wrote: Ah. Python's probably not a great idea either, since we'd still have to transcribe all the tests and in the end we'd land in a non-JVM language.. In that case it might be worth revisiting bazel-integration-testing, since we'd be able to keep the tests in a JVM language that way. (Quick aside though, we ran into some early issues: bazelbuild/bazel-integration-testing#153 <bazelbuild/bazel-integration-testing#153>) As an *intermediate* step, however, I still think it would be worthwhile to break up test_rules_scala.sh. We'll need to do this regardless, and migrating individual tests will be less daunting than trying to do them all at once. I suggest we break it up into small shell scripts as the first step and then migrate those to whichever framework we decide on. This will also give us the benefit of being able to run individual tests sooner, while we migrate to something like bazel-integration-testing. Does this sound reasonable @ittaiz <https://github.com/ittaiz> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#824?email_source=notifications&email_token=AAKQQF7OVAUYHHL2DMEXFPTQKFPELA5CNFSM4INE526KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD66FCHI#issuecomment-532435229>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKQQF5UDU5RH5MTKUOLX2DQKFPELANCNFSM4INE526A> .

-- *Ittai Zeidman* Cell: 054-6735021 40 Hanamal street, Tel Aviv, Israel <http://www.wix.com>

[borkaehw]

I believe a refactoring of splitting up tests into separate file could be a first step toward our goal. I have a draft PR open #849.