Parallelized Travis runs split test, coverage reports

[jywarren]

so we get partial reports of everything, like our coverage being reported at 53%, which is actually only the coverage calculated from the system tests task: https://travis-ci.org/publiclab/plots2/jobs/548708932#L4077

In a similar vein, our PR Dangerbot posts can no longer access aggregate junit reports in Writing XML reports to /app/test/reports -- since we're running different tests in parallel in each of 5 tasks (and 5 different VMs).

I think we have to brainstorm on this -- mostly for our coverage reporting but also for the Dangerbot failure reporting, which is now erroring and not linking into any tests.

plots2/Dangerfile

Lines 32 to 44 in 21261a5

	junit.parse "output.xml"
	junit.failures.collect(&:nodes).flatten.each do |failure|
	failure.nodes.each do |f|
	f.match(/(test[a-z_\/]+.rb):([0-9]+)/) do |match|
	source_path = match[1]
	line = match[2]
	if !source_path.nil? && !line.nil?
	f = f.gsub(source_path + ':' + line, "<a href='https://github.com/#{github.pr_author}/plots2/tree/#{github.branch_for_head}/#{source_path}#L#{line}'>#{source_path}:#{line}</a>")
	end
	fail("There was a test failure at: #{f}")
	end
	end
	end

[kaustubh-nair]

We're facing a similar issue in mapknitter as well. Codecov reports a coverage of 40% which is only due to unit tests. I believe we need to find a way to accumulate all the reports and send them together to Codecov. Instead of sending 5 different reports
CC @alaxalves

[jywarren]

Is there any shared storage of any kind between tasks?

…

On Tue, Jun 25, 2019 at 3:21 AM Kaustubh Nair ***@***.***> wrote: We're facing a similar issue in mapknitter as well. Codecov reports a coverage of 40% which is only due to unit tests. I believe we need to find a way to accumulate all the reports and send them together to Codecov. Instead of sending 5 different reports CC @alaxalves <https://github.com/alaxalves> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5931?email_source=notifications&email_token=AAAF6J27Q3QCPOXZOWNIHZTP4HBO7A5CNFSM4H2SKZP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYPJIMQ#issuecomment-505320498>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAF6JZAIPKOBSBPYI5VTMLP4HBO7ANCNFSM4H2SKZPQ> .

[alaxalves]

Is there any shared storage of any kind between tasks?
…
On Tue, Jun 25, 2019 at 3:21 AM Kaustubh Nair @.***> wrote: We're facing a similar issue in mapknitter as well. Codecov reports a coverage of 40% which is only due to unit tests. I believe we need to find a way to accumulate all the reports and send them together to Codecov. Instead of sending 5 different reports CC @alaxalves https://github.com/alaxalves — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5931?email_source=notifications&email_token=AAAF6J27Q3QCPOXZOWNIHZTP4HBO7A5CNFSM4H2SKZP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYPJIMQ#issuecomment-505320498>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6JZAIPKOBSBPYI5VTMLP4HBO7ANCNFSM4H2SKZPQ .

I'll into it. I'm already digging this to Mapknitter, when I get it figured out there, I'll do it here as well.

[jywarren]

Hm. Thank you both! Note in relation to this one, I'm using an env var in #5929 to make only one of the 5 tasks actually generate a Danger report!

[kaustubh-nair]

@jywarren are we using coveralls for plots2?
If so, we can simply set COVERALLS_PARALLEL=true for merging the reports.
http://docs.coveralls.io/parallel-build-webhook

[kaustubh-nair]

And set up a webhook

[kaustubh-nair]

It would be easier to switch to Codecov. They auto merge test reports if we use the gem. I've set it up here publiclab/mapknitter#749
Thanks!

[jywarren]

Let's switch to codecov! It's much nicer! Thanks so much!

…

On Wed, Jun 26, 2019, 2:28 PM Kaustubh Nair ***@***.***> wrote: It would be easier to switch to Codecov. They support auto merge test reports if we use the gem. I've set it up here publiclab/mapknitter#749 <publiclab/mapknitter#749> Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5931?email_source=notifications&email_token=AAAF6JZP2VSIK2MTAG4FQOLP4OYKTA5CNFSM4H2SKZP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYUNJRY#issuecomment-505992391>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAF6J3Q7Q4F4JVAAIXQZO3P4OYKTANCNFSM4H2SKZPQ> .

[kaustubh-nair]

Okay! I'll submit a PR if you'd like

[jywarren]

that would be super. thank you!

…

On Wed, Jun 26, 2019 at 2:50 PM Kaustubh Nair ***@***.***> wrote: Okay! I'll submit a PR if you'd like — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5931?email_source=notifications&email_token=AAAF6J55Y56L3SE46ST4XLDP4O25VA5CNFSM4H2SKZP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYUPIGQ#issuecomment-506000410>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAF6J77FCTVCT4KDIPBFLLP4O25VANCNFSM4H2SKZPQ> .