[CI] Split Integration tests out of first phase of pipeline

[Mousius]

I took a look at the time taken by each stage in the Jenkins pipeline and what comprises the 6 hour CI build time. CPU Integration tests took 60 minutes of the 100 minutes of Build: CPU. By adding python3: CPU with just those Integration tests, it lines up with python3: GPU and python3: i386 which both take a similar amount of time and takes roughly 60 minutes off the overall run time.

Numbers copied from sample successful run (final time approx: 358 minutes):

Phase	ID	Job	Minutes	Start
0	0	Sanity	3	0
1	0	BUILD: arm	2	3
1	1	BUILD: i386	33	3
1	2	BUILD: CPU	100	3
1	3	BUILD: GPU	25	3
1	4	BUILD: QEMU	6	3
1	5	BUILD: WASM	2	3
2	0	java: GPU	1	103
2	1	python3: GPU	66	103
2	2	python3: arm	22	103
2	3	python3: i386	70	103
3	0	docs: GPU	3	173
3	1	frontend: CPU	40	173
3	2	frontend: GPU	185	173
3	3	topi: GPU	110	173

Numbers predicted after change (final time approx: 293 minutes):

Phase	ID	Job	Minutes	Start
0	0	Sanity	3	0
1	0	BUILD: arm	2	3
1	1	BUILD: i386	33	3
1	2	BUILD: CPU	35	3
1	3	BUILD: GPU	25	3
1	4	BUILD: QEMU	6	3
1	5	BUILD: WASM	2	3
2	0	java: GPU	1	38
2	1	python3: GPU	66	38
2	2	python3: arm	22	38
2	3	python3: i386	70	38
2	4	python3: CPU	60	38
3	0	docs: GPU	3	108
3	1	frontend: CPU	40	108
3	2	frontend: GPU	185	108
3	3	topi: GPU	110	108

[areusch]

@tqchen can you comment on why we have integration tests in the first part? IIRC it was originally due to scarcity of GPU nodes but now perhaps we don't need to worry so much. wdyt? i agree with @Mousius assessment that the CPU is the long pole in the first phase, and switching to xdist will only make that more obvious.

[tqchen]

i agree we can do that, the main thing is to be able to test on staging before merge

[Mousius]

Thanks for taking a look @areusch / @tqchen - could someone trigger the staging job so we can see if I've managed to get the incantation right? 😸

[jroesch]

This is building, https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/160/pipeline only merge after this has gone green.

[jroesch]

Wait for CI staging run before merging.

[Mousius]

Had to re-push due to a flakey unit test on the PR build, docker-staging build is here now:
https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/161/pipeline

[jroesch]

In bias for action, I am going to merge this one feel free to follow up if other things need to happen.