sharness: Generate JUnit test reports #4530

magik6k · 2017-12-28T23:09:15Z

This makes sharness generate a JUnit XML report that can be used by Jenkins to view test results in a bit nicer form than a huge txt file (removing the fun of wasting few minutes to find the failure each time one happens). The report is saved as test/sharness/test-results/sharness.xml.

Example report: https://ipfs.io/ipfs/QmZVCjX2L9DrbqYaVeVzTXZZUbyQvTM3mTUmzeMCoN8Pqc/sharness.xml - HTML from junit2html - Jenkins should show failures at the top (like here)

Somewhat depends on ipfs-inactive/jenkins#85

whyrusleeping · 2017-12-31T20:57:51Z

(removing the fun of wasting few minutes to find the failure each time one happens)

Aww, but that was my favorite part

whyrusleeping · 2018-01-04T20:07:26Z

@magik6k @victorbjelkholm whats the blocker here?

magik6k · 2018-01-04T20:31:41Z

I'd like to have test result collection enabled on jenkins before this is merged to make sure it works well with our setup (though I have checked it with local jenkins instance and it did work).

That's done in ipfs-inactive/jenkins#85, though @victorbjelkholm suggests to defer/integrate those changes to #4430 / ipfs-inactive/jenkins#81 (Which will probably take more time, but it needs to be done either way)

TL;DR stuff in this PR Works For Me(tm), I'd like to see it not breaking production Jenkins before it's merged

magik6k · 2018-03-13T20:11:12Z

tbh I'd love to have this enabled already, even with the legacy job as it doesn't appear the new one is getting enabled anytime soon, and looking through failed sharness output is getting annoying.

I rebased this, shouldn't break anything, some review would be welcome.

Kubuxu · 2018-03-14T18:11:02Z

test/sharness/Rules.mk

 .PHONY: $(T_$(d))

 $(d)/aggregate: $(T_$(d))
 	@echo "*** $@ ***"
 	@(cd $(@D) && ./lib/test-aggregate-results.sh)
+	@(cd $(@D) && ./lib/gen-junit-report.sh)


I would make this separate target that corresponds to created file.

Kubuxu · 2018-03-14T18:43:17Z

test/sharness/Rules.mk

 .PHONY: $(T_$(d))

 $(d)/aggregate: $(T_$(d))
 	@echo "*** $@ ***"
 	@(cd $(@D) && ./lib/test-aggregate-results.sh)
 .PHONY: $(d)/aggregate

+$(d)/junit_xml: $(T_$(d))


I meant, name it test-results/sharness.xml as it produces this file.

Kubuxu · 2018-03-14T18:43:46Z

test/sharness/Rules.mk

@@ -52,7 +61,7 @@ $(d)/deps: $(SHARNESS_$(d)) $$(DEPS_$(d)) # use second expansion so coverage can
 test_sharness_deps: $(d)/deps
 .PHONY: test_sharness_deps

-test_sharness_short: $(d)/aggregate
+test_sharness_short: $(d)/aggregate $(d)/junit_xml


Most users won't need the JUnit xmls so there is no reason to create them for them.

Kubuxu · 2018-03-14T18:44:37Z

test/sharness/lib/0001-Generate-partial-JUnit-reports.patch

+ 	test $total_prereq = $ok_prereq
+ }
+
+junit_testcase() {


Can we have it noop when JUnit is not needed? It is file IO, it can be slow.

Kubuxu · 2018-03-14T18:44:45Z

test/sharness/lib/0001-Generate-partial-JUnit-reports.patch

+ 		test -z "$1" && test -n "$quiet" && return
+ 		shift
+
+		echo "$*" >> .junit/tout


Same in here.

Kubuxu · 2018-03-14T18:44:52Z

test/sharness/lib/0001-Generate-partial-JUnit-reports.patch

+ 		esac
+ 		shift
+
+		echo "$*" >> .junit/tout


Kubuxu · 2018-03-14T18:45:09Z

test/sharness/lib/0001-Generate-partial-JUnit-reports.patch

+ 	# This is a separate function because some tests use
+ 	# "return" to end a test_expect_success block early.
+-	eval </dev/null >&3 2>&4 "$*"
+	eval </dev/null > >(tee -a .junit/tout >&3) 2> >(tee -a .junit/terr >&4) "$*"


and in general.

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

Kubuxu · 2018-04-11T01:44:39Z

@magik6k can we not increase the scope of this PR with benchmarking? Let's get it merged ASAP.

magik6k · 2018-04-11T11:28:07Z

Good point. Extracted the benchmark part into #4932

Kubuxu · 2018-04-19T11:57:06Z

@magik6k can we resolve the Jenkins being red? I am ok with even disabling OSX and Windows on Jenkins for time being until we can resolve it.

The JUnit reporting is important enough to have it merged before that can be resolved.

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

magik6k · 2018-04-19T16:50:11Z

It will be resolved by ipfs-inactive/jenkins#107, which i hope will be resolved soon

Kubuxu · 2018-04-20T14:40:14Z

ipfs-inactive/jenkins#107 is resolved. RFM?

magik6k added the kind/test Testing work label Dec 28, 2017

ghost assigned magik6k Dec 28, 2017

ghost added the status/in-progress In progress label Dec 28, 2017

magik6k mentioned this pull request Dec 29, 2017

Collect junit test reports for go-ipfs sharness ipfs-inactive/jenkins#85

Closed

magik6k added the status/blocked Unable to be worked further until needs are met label Dec 29, 2017

magik6k force-pushed the feat/sharness-junit branch from bb85024 to cfe0d29 Compare March 12, 2018 20:26

magik6k requested a review from Kubuxu as a code owner March 12, 2018 20:26

magik6k force-pushed the feat/sharness-junit branch from cfe0d29 to e6cbe87 Compare March 12, 2018 20:41

magik6k removed the status/blocked Unable to be worked further until needs are met label Mar 13, 2018

magik6k added the need/review Needs a review label Mar 14, 2018

Kubuxu reviewed Mar 14, 2018

View reviewed changes

test/sharness/lib/0001-Generate-partial-JUnit-reports.patch Outdated

esac

shift

+

+ echo "$*" >> .junit/tout

Copy link

Member

Kubuxu Mar 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And here.

Kubuxu reviewed Mar 14, 2018

View reviewed changes

magik6k force-pushed the feat/sharness-junit branch 9 times, most recently from 059a80a to 087e114 Compare March 14, 2018 22:32

magik6k added 3 commits April 10, 2018 18:31

sharness: Don't escape colors in junit xml

3e41b15

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

ci: Make sure to fetch deps on go build

9a3ff19

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

ci: run go vet, disable docker on jenkins

6b3915b

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

magik6k force-pushed the feat/sharness-junit branch from 934e5f0 to 826fcab Compare April 10, 2018 16:31

magik6k force-pushed the feat/sharness-junit branch from 826fcab to 6b3915b Compare April 11, 2018 11:21

magik6k mentioned this pull request Apr 11, 2018

[WIP] Run benchmarks on Jenkins #4932

Closed

magik6k force-pushed the feat/sharness-junit branch 9 times, most recently from d867e18 to 7b6c449 Compare April 19, 2018 11:44

magik6k force-pushed the feat/sharness-junit branch 2 times, most recently from faceaf6 to 272718c Compare April 19, 2018 13:04

ci: ignore windows/mac test results on jenkins

fe62f77

License: MIT Signed-off-by: Łukasz Magiera <[email protected]>

magik6k force-pushed the feat/sharness-junit branch from 272718c to fe62f77 Compare April 19, 2018 16:35

Kubuxu added RFM and removed need/review Needs a review labels Apr 20, 2018

Kubuxu approved these changes Apr 20, 2018

View reviewed changes

whyrusleeping merged commit 67a8cb0 into master Apr 20, 2018

ghost removed the status/in-progress In progress label Apr 20, 2018

whyrusleeping deleted the feat/sharness-junit branch April 20, 2018 14:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sharness: Generate JUnit test reports #4530

sharness: Generate JUnit test reports #4530

magik6k commented Dec 28, 2017 •

edited

Loading

whyrusleeping commented Dec 31, 2017

whyrusleeping commented Jan 4, 2018

magik6k commented Jan 4, 2018

magik6k commented Mar 13, 2018

Kubuxu Mar 14, 2018

Kubuxu Mar 14, 2018

Kubuxu Mar 14, 2018

Kubuxu Mar 14, 2018

Kubuxu Mar 14, 2018

Kubuxu Mar 14, 2018

Kubuxu Mar 14, 2018

Kubuxu Mar 14, 2018

Kubuxu commented Apr 11, 2018

magik6k commented Apr 11, 2018

Kubuxu commented Apr 19, 2018

magik6k commented Apr 19, 2018

Kubuxu commented Apr 20, 2018

sharness: Generate JUnit test reports #4530

sharness: Generate JUnit test reports #4530

Conversation

magik6k commented Dec 28, 2017 • edited Loading

whyrusleeping commented Dec 31, 2017

whyrusleeping commented Jan 4, 2018

magik6k commented Jan 4, 2018

magik6k commented Mar 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kubuxu commented Apr 11, 2018

magik6k commented Apr 11, 2018

Kubuxu commented Apr 19, 2018

magik6k commented Apr 19, 2018

Kubuxu commented Apr 20, 2018

magik6k commented Dec 28, 2017 •

edited

Loading