--split doesn't work in Chrome

[lolmaus]

This issue may or may not be related to using Chrome for testing (e. g. with new Ember CLI defaults).

Here I have a simple repo with two acceptance test files, three tests each: https://github.com/lolmaus/ember-exam-issue-108

ember exam --split 2

Click here to expand console output

'instrument' is imported from external module 'ember-data/-debug' but never used
Warning: ignoring input sourcemap for vendor/ember/ember.debug.js because ENOENT: no such file or directory, open '/home/lolmaus/Code/ember-exam-chrome-problem/tmp/source_map_concat-input_base_path-Y4zKpLr8.tmp/vendor/ember/ember.debug.map'
Warning: ignoring input sourcemap for vendor/ember/ember-testing.js because ENOENT: no such file or directory, open '/home/lolmaus/Code/ember-exam-chrome-problem/tmp/source_map_concat-input_base_path-n7B48q2W.tmp/vendor/ember/ember-testing.map'
cleaning up...
Built project successfully. Stored in "/home/lolmaus/Code/ember-exam-chrome-problem/tmp/class-tests_dist-twKSuYKz.tmp".
ok 1 Chrome 60.0 - Exam Partition #1 - Acceptance | bar: bar 1
ok 2 Chrome 60.0 - Exam Partition #1 - Acceptance | bar: bar 2
ok 3 Chrome 60.0 - Exam Partition #1 - Acceptance | bar: bar 3
ok 4 Chrome 60.0 - Exam Partition #1 - Acceptance | foo: foo 1
ok 5 Chrome 60.0 - Exam Partition #1 - Acceptance | foo: foo 2
ok 6 Chrome 60.0 - Exam Partition #1 - Acceptance | foo: foo 3
ok 7 Chrome 60.0 - Exam Partition #1 - ESLint | app: app.js
ok 8 Chrome 60.0 - Exam Partition #1 - ESLint | app: resolver.js
ok 9 Chrome 60.0 - Exam Partition #1 - ESLint | app: router.js
ok 10 Chrome 60.0 - Exam Partition #1 - ESLint | tests: acceptance/bar-test.js
ok 11 Chrome 60.0 - Exam Partition #1 - ESLint | tests: acceptance/foo-test.js
ok 12 Chrome 60.0 - Exam Partition #1 - ESLint | tests: helpers/destroy-app.js
ok 13 Chrome 60.0 - Exam Partition #1 - ESLint | tests: helpers/module-for-acceptance.js
ok 14 Chrome 60.0 - Exam Partition #1 - ESLint | tests: helpers/pause.js
ok 15 Chrome 60.0 - Exam Partition #1 - ESLint | tests: helpers/resolver.js
ok 16 Chrome 60.0 - Exam Partition #1 - ESLint | tests: helpers/start-app.js
ok 17 Chrome 60.0 - Exam Partition #1 - ESLint | tests: test-helper.js

1..17
# tests 17
# pass  17
# skip  0
# fail  0

# ok

As you can see, all tests run in a single partition.

ember exam --split 2 --parallel

Click here to expand console output

'instrument' is imported from external module 'ember-data/-debug' but never used
Warning: ignoring input sourcemap for vendor/ember/ember.debug.js because ENOENT: no such file or directory, open '/home/lolmaus/Code/ember-exam-chrome-problem/tmp/source_map_concat-input_base_path-02s2KTMk.tmp/vendor/ember/ember.debug.map'
Warning: ignoring input sourcemap for vendor/ember/ember-testing.js because ENOENT: no such file or directory, open '/home/lolmaus/Code/ember-exam-chrome-problem/tmp/source_map_concat-input_base_path-vRKCgsNm.tmp/vendor/ember/ember-testing.map'
cleaning up...
Built project successfully. Stored in "/home/lolmaus/Code/ember-exam-chrome-problem/tmp/class-tests_dist-Hw4KL0Nq.tmp".
ok 1 Chrome 60.0 - Exam Partition #2 - Acceptance | bar: bar 1
ok 2 Chrome 60.0 - Exam Partition #1 - Acceptance | bar: bar 1
ok 3 Chrome 60.0 - Exam Partition #2 - Acceptance | bar: bar 2
ok 4 Chrome 60.0 - Exam Partition #1 - Acceptance | bar: bar 2
ok 5 Chrome 60.0 - Exam Partition #2 - Acceptance | bar: bar 3
ok 6 Chrome 60.0 - Exam Partition #1 - Acceptance | bar: bar 3
ok 7 Chrome 60.0 - Exam Partition #2 - Acceptance | foo: foo 1
ok 8 Chrome 60.0 - Exam Partition #1 - Acceptance | foo: foo 1
ok 9 Chrome 60.0 - Exam Partition #2 - Acceptance | foo: foo 2
ok 10 Chrome 60.0 - Exam Partition #1 - Acceptance | foo: foo 2
ok 11 Chrome 60.0 - Exam Partition #2 - Acceptance | foo: foo 3
ok 12 Chrome 60.0 - Exam Partition #1 - Acceptance | foo: foo 3
ok 13 Chrome 60.0 - Exam Partition #2 - ESLint | app: app.js
ok 14 Chrome 60.0 - Exam Partition #1 - ESLint | app: app.js
ok 15 Chrome 60.0 - Exam Partition #2 - ESLint | app: resolver.js
ok 16 Chrome 60.0 - Exam Partition #1 - ESLint | app: resolver.js
ok 17 Chrome 60.0 - Exam Partition #2 - ESLint | app: router.js
ok 18 Chrome 60.0 - Exam Partition #1 - ESLint | app: router.js
ok 19 Chrome 60.0 - Exam Partition #2 - ESLint | tests: acceptance/bar-test.js
ok 20 Chrome 60.0 - Exam Partition #1 - ESLint | tests: acceptance/bar-test.js
ok 21 Chrome 60.0 - Exam Partition #2 - ESLint | tests: acceptance/foo-test.js
ok 22 Chrome 60.0 - Exam Partition #1 - ESLint | tests: acceptance/foo-test.js
ok 23 Chrome 60.0 - Exam Partition #2 - ESLint | tests: helpers/destroy-app.js
ok 24 Chrome 60.0 - Exam Partition #1 - ESLint | tests: helpers/destroy-app.js
ok 25 Chrome 60.0 - Exam Partition #2 - ESLint | tests: helpers/module-for-acceptance.js
ok 26 Chrome 60.0 - Exam Partition #1 - ESLint | tests: helpers/module-for-acceptance.js
ok 27 Chrome 60.0 - Exam Partition #2 - ESLint | tests: helpers/pause.js
ok 28 Chrome 60.0 - Exam Partition #1 - ESLint | tests: helpers/pause.js
ok 29 Chrome 60.0 - Exam Partition #2 - ESLint | tests: helpers/resolver.js
ok 30 Chrome 60.0 - Exam Partition #1 - ESLint | tests: helpers/resolver.js
ok 31 Chrome 60.0 - Exam Partition #2 - ESLint | tests: helpers/start-app.js
ok 32 Chrome 60.0 - Exam Partition #1 - ESLint | tests: helpers/start-app.js
ok 33 Chrome 60.0 - Exam Partition #2 - ESLint | tests: test-helper.js
ok 34 Chrome 60.0 - Exam Partition #1 - ESLint | tests: test-helper.js

1..34
# tests 34
# pass  34
# skip  0
# fail  0

# ok

Adding --parallel makes the requested number of partitions to appear, but there are two other issues, but every partition has all tests inside it. In other words, tehre are duplicates: the total number of executed tests is the total number of defined tests multiplied by the number of partitions.

---

Tested in Linux Mint 18.2 (based on Ubuntu 16.04). We see the same behavior in our CentOS-driven CI server.

[lolmaus]

Updated with link to reproduction repo.

[lolmaus]

CC @vvainio.

[andrewzamojc-pd]

I needed to add a "parallel" option to my testem.js config file, and only "-1" (max workers) worked well for me. Without letting testem know about the parallelization, I would see every partition running every test. The behaviour you described without the option still sounds pretty odd, but maybe this can get your tests running in parallel at least.

Ember Exam will respect the parallel setting of your Testem config file while running tests in parallel. Note that the default value for parallel in Testem is 1, which means you'll need a non-default value to actually see parallel behavior.

https://github.com/trentmwillis/ember-exam

You can even set the number of parallel workers manually or set it to -1 to have testem max out whatever your CI server can handle

https://til.hashrocket.com/posts/3fc396bee6-testem-supports-parallel-execution

[lolmaus]

@andrewzamojc-pd Ah, thank you. I totally missed configuring parallel in testem.js. Fixed that now.

Issue description updated.

[andrewzamojc-pd]

@lolmaus So parallel 4 doesn't fix your issue? Could you try parallel "-1" (max workers possible)? That's the only value that worked as expected for me, and if that works for you it could be useful for finding the root cause.

[lolmaus]

When I added parallel: 4 to testem.js, I saw a difference: the tests started running in parallel. It is visible in the log. I've just updated the issue title to reflect that.

But the splitting problem still stands.

[andrewzamojc-pd]

did you try parallel: -1? It may fix the splitting problem.

[lolmaus]

@andrewzamojc-pd It does not. I've updated the demo repo with -1. I encourage you run it and see or, even better, try it yourself in any project driven by Ember CLI 2.15+.

[andrewzamojc-pd]

Cool, just wanted to make sure since it was only -1 that worked for me quite recently, although in an Ember cli 2.12 project :S I'll give your project a whirl.

[begedin]

Just to add to the conversation, in our project, the same thing seems to be happening. Adding parallel: -1 to testem.js does make our 3 partitions run in parallel, but each partition still executes ALL of the tests in the suite, not just a fraction of them.

[lolmaus]

@trentmwillis Any chance you look into this?

[trentmwillis]

Sorry, been busy/on vacation. Taking a brief look at the repo, looks like you need to call the loadEmberExam function in this file.

Based on what I gather from above, I think that should fix it.

[lolmaus]

Thank you for looking into this, @trentmwillis. 🙇

We're currently blocked by Ember Data: emberjs/data#5136, emberjs/data#5111, emberjs/data#5175. Once the upgrade path is clear, we will employ this fix.

Meanwhile, have you considered adding a blueprint to update testem.js and tests/test-helper.js automatically on install?

[trentmwillis]

Okay, I'm going to close this as I believe the issue here is resolved.

Meanwhile, have you considered adding a blueprint to update testem.js and tests/test-helper.js automatically on install?

I have not. I'm not opposed to it, if you would like to do the work for it, but overall I don't feel it would add much value for the amount of code needed for it to be a robust solution (since Testem config could also be testem.json, .testem.json, testem.yml, or .testem.yml). If you want to pursue, feel free to open another issue to track it.

In the interim, I'm going to updates the docs so that initial setup is clearer.