CI Jobs are timing out in the restructured-pipeline

[DonJayamanne]

With the new changes we're running functional tests and single root worksapce tests in the same job.
Unfortunately both are slow on windows.

• Functional tests >10 minutes
• Single root workspace tests ~= 50 minutes
• Job has a timeout of 60 minutes

Hence the jobs are cancelled before completion.

We'd need to split them into two separate jobs or increase the timeout.
Increasing timeout only causes the entire pipeline to run slower.
I.e. two separate jobs results in parallelized jobs = faster.

Bug as this now causes jobs to fail cancel out, which didn't in the past.
Also, if there are failures in the tests, these won't be reported anymore, as the whole job has been cancelled

[karthiknadig]

May be we should also increase the Max Parallel. Currently it is 2, and it should be more.

[DonJayamanne]

May be we should also increase the Max Parallel. Currently it is 2, and it should be more.

Sure, i'll create a separate PR, or we need a new issue for that fix.

[DonJayamanne]

Oh sorry, @karthiknadig that's fine, I though this was my PR, hence asked for a separate issue for the change.

[ericsnowcurrently]

For the PR pipeline it would be better if we could increase the timeout instead of splitting the job up. The job is not a required one so folks don't need to wait for it if the PR changes don't necessitate that level of validation. If we split the job then that is 1 extra build agent we are using every time a PR gets created or updated.

For the other pipelines (not sure they are affected), splitting it would be fine.

[ericsnowcurrently]

May be we should also increase the Max Parallel. Currently it is 2, and it should be more.

Keep in mind that increasing the max parallel setting will mean we are using more agents, which is what we were trying to improve.

Also, increasing the max parallel value shouldn't impact timeouts, so I don't see the value in doing so. If folks are running the CI (or other) pipeline manually and that is taking too long then we should have a separate manual-only pipeline with a higher max parallel value. In fact, I already added a build/ci/vscode-python-ci-manual.yaml and it only needs to be set up on AzDO.

[ericsnowcurrently]

As I mentioned elsewhere, I'm sure that splitting the job back into 2 would be fine.

FWIW, my only concern is that the repo has any PR-gating CI jobs that take more than a few minutes to run. At some point we need to have our unit test coverage to the level that we can confidently rely on it almost exclusively for gating PRs (with perhaps a handful of fast functional tests to cover OS-specific behaviors). However, that's a discussion for a different forum. 😄

[DonJayamanne]

Validated