[workflows/linux] Switch to self-hosted runners in Azure Kubernetes Service #19183

ababushk · 2023-08-14T15:10:16Z

Details:

This PR aims to move Linux builds to self-hosted runners in Azure Kubernetes Service. This will remove our dependencies on the limited number of default GitHub Runners ran in parallel.

Tickets:

117508

I want to test worst-case scenario

.github/workflows/linux.yml

ababushk · 2023-09-18T19:27:47Z

The paths were hard-coded because these issues:

github.workspace and runner.workspace are incorrect inside container jobs actions/runner#2058 - variables that use ${{ github.workspace }} have a value that points to non-existent path inside the container, but if we use shell expansion instead of ${{ env.FOOBAR }}, it works.
[bug] Only first artifact is found when using upload-artifact@v3 inside a container actions/upload-artifact#441 upload-artifacts@v3 translates only the first path provided to a path that exists inside the container

ababushk · 2023-09-18T19:29:48Z

concurrency has also been changed. @mryzhov and me cancelled each other's jobs with the old value.

.github/workflows/linux.yml

Co-authored-by: Ilya Lavrenov <[email protected]>

mryzhov · 2023-09-22T12:50:32Z

Regarding increased execution time:
cpu_func_tests silently skip some tests depending on hardware

ilya-lavrenov · 2023-09-22T12:50:17Z

.github/workflows/linux.yml

@@ -882,6 +921,7 @@ jobs:
        shell: bash
    runs-on: ${{ github.event_name == 'schedule' && 'ubuntu-20.04-8-cores' || 'ubuntu-20.04'}}
    env:
+      DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input


do we need it here?
dependencies are not installed in this job

I do not undestand - why was it added for PyTorch_Models_Tests but not for TensorFlow_Hub_Models_Tests?
May be you missed:)

it's not required in both cases, because we don't install apt packages in these jobs

…ervice (openvinotoolkit#19183) * [workflows/liux] Trying AKS runners * Remove sudo usages * Fix repositories path * Well, adding debug output * execution bit? * more debug output * using bash... * We should use shell syntax for env variables in 'run' sections * again, may be we should use bash... * fix cloning path for openvino repo * more debug output * Trying to populate WORKSPACE var in other way * Trying different WORKSPACE * export ALL variables for samples tests * Trying to put openvino build into correct directory * Replace more occurences of BUILD_DIR... * Use shell expansion instead of Actions syntax in run steps * Skip unit test failures * Oops, there's no pytest for MO unit tests * Oops, typo * Fix LAYER_TESTS_INSTALL_DIR variable expansion * Fix LAYER_TESTS_INSTALL_DIR for new stages * Replace all the varibles * Comment out CCACHE_DIR I want to test worst-case scenario * Comment setup ccache step * Revert "Comment out CCACHE_DIR" This reverts commit 046663a. * Just checking something * Update with new stages from master * One more stage to fix * Fix INSTALL_DIR again (new stages were merged to master?) * Remove concurrency limit (for test purposes) * Trigger one more pipeline * Disable concurrency settings * Trigger another build * Disable ccache (for testing purposes) * trigger another build to measure duration without ccache * And one more test commit * Return back ccache * Testing new image * Enable concurrency back * Using new image * And another image * Trying build on the clean image * Install git Upstream ubuntu image doesn't have git * Oops, forgot apt-get update * Upstream Ubuntu image, but using ACR cache * Remove debug output * Use Ubuntu image from ACR cache for the rest of the stages * Display version of ccache * Swith to ubuntu 20 temporarily (I hope) * Set CCACHE_TEMP_DIR outside of shared space * Revert "Swith to ubuntu 20 temporarily (I hope)" This reverts commit 2e53109. * trying ccche nohardlink option * Trying to disable ccache file cloning * fix CCACHE_TEMPDIR * Install git to Python unit tests stage * Install libsnappy-dev as tests require it * Install python to test stage * Fix env vars syntax in few places * Update docker image name * Increase timeout for CPU tests * Debug output - trying to find out what's wrong with test reports * Change file name in debug-output 'find' command * let's try shell syntax for upload artifact action... It most likely won't work but I see strange output in the logs: Error: No files were found with the provided path: /__w/openvino/openvino/install/tests/TEST*.xml /runner/_work/openvino/openvino/install/tests/logs/failed/*.log /runner/_work/openvino/openvino/install/tests/logs/crashed/*.log /runner/_work/openvino/openvino/install/tests/logs/hanged/*.log /runner/_work/openvino/openvino/install/tests/logs/interapted/*.log /runner/_work/openvino/openvino/install/tests/logs/disabled_tests.log. No artifacts will be uploaded. So the first path is translated to the path inside the container correctly and the rest are not O_o * Revert "let's try shell syntax for upload artifact action..." This reverts commit eb19b4f. * Leave just disabled_tests.log (as it is the only one present) * Use my own directory instead of github.workspace Inside-container path translation doesn't work for upload-artifact actions (it works only for the first path, but not for the rest) * Revert "Leave just disabled_tests.log (as it is the only one present)" This reverts commit 4321827. * Okay, let's try hardcoded workspace path... * Okay, path should be inside "official" workspace. Hardcode it * Missed some variables, replace github.workspace to hard-coded value * Remove debug output 'find' command * Remove old-style cache setup * Clean ccache stats after cmake * Change docker image to the one with non-root user * Oops, switch to Ubuntu 20 * Playing with concurrency group name * Oops, forgot sudo for ninja installation * Revert "Oops, forgot sudo for ninja installation" This reverts commit a7e1730. * Revert "Oops, switch to Ubuntu 20" This reverts commit 19fe75f. * Revert "Change docker image to the one with non-root user" This reverts commit 5531164. * Use Ubuntu 20 DockerHub image * Trying to force apt-get packages not to wait for user input * Make DEBIAN_FRONTED for all stages * Update reason of skipped MO unit test * Add DEBIAN_FRONTED to all jobs * Fix environment variables expansion in newly-added stages * Update .github/workflows/linux.yml Co-authored-by: Ilya Lavrenov <[email protected]> * Update .github/workflows/linux.yml Co-authored-by: Ilya Lavrenov <[email protected]> * Fix review comments * More review comment fixes * Remove DEBIAN_FRONTEND from Python Models job * "Get tools versions" step is absent on master --------- Co-authored-by: Ilya Lavrenov <[email protected]>

[workflows/liux] Trying AKS runners

98fa53f

github-actions bot added the category: CI OpenVINO public CI label Aug 14, 2023

ababushk added 7 commits August 14, 2023 16:16

Remove sudo usages

b521458

Fix repositories path

e5d2c97

Well, adding debug output

86e8272

execution bit?

3b9ae1b

more debug output

2b01b97

using bash...

27dbef4

We should use shell syntax for env variables in 'run' sections

89966e7

ababushk force-pushed the aks_runners branch from be28bdf to 89966e7 Compare August 14, 2023 16:45

ababushk added 5 commits August 14, 2023 18:41

again, may be we should use bash...

be5c572

fix cloning path for openvino repo

f7292e8

more debug output

1e92504

Trying to populate WORKSPACE var in other way

9fe2092

Trying different WORKSPACE

8414e25

openvinotoolkit deleted a comment from sys-openvino-ci Aug 16, 2023

ababushk added 6 commits August 16, 2023 18:12

export ALL variables for samples tests

bee1434

Trying to put openvino build into correct directory

d56336f

Replace more occurences of BUILD_DIR...

4aeb2b6

Use shell expansion instead of Actions syntax in run steps

1d3706f

Merge branch 'master' into aks_runners

a564e2f

Skip unit test failures

a26a13c

github-actions bot added the category: MO Model Optimizer label Aug 21, 2023

ababushk added 7 commits August 21, 2023 21:10

Oops, there's no pytest for MO unit tests

f0d960a

Oops, typo

2e499d1

Fix LAYER_TESTS_INSTALL_DIR variable expansion

c708832

Merge branch 'master' into aks_runners

54776ea

Fix LAYER_TESTS_INSTALL_DIR for new stages

529744d

Replace all the varibles

b7109d6

Comment out CCACHE_DIR

046663a

I want to test worst-case scenario

ababushk added 2 commits September 15, 2023 18:43

Fix environment variables expansion in newly-added stages

20628b0

Merge branch 'master' into aks_runners

abda3f1

github-actions bot added the github_actions Pull requests that update GitHub Actions code label Sep 18, 2023

mryzhov reviewed Sep 18, 2023

View reviewed changes

.github/workflows/linux.yml Show resolved Hide resolved

ababushk marked this pull request as ready for review September 18, 2023 19:27

ababushk requested review from a team as code owners September 18, 2023 19:27

Merge branch 'master' into aks_runners

69b9ffd

Merge branch 'master' into aks_runners

3f46058

ababushk changed the title ~~[workflows/linux] Trying AKS runners~~ [workflows/linux] Switch to self-hosted runners in Azure Kubernetes Service Sep 19, 2023

mryzhov reviewed Sep 19, 2023

View reviewed changes

.github/workflows/linux.yml Show resolved Hide resolved

Merge branch 'master' into aks_runners

aae00d5

jane-intel approved these changes Sep 20, 2023

View reviewed changes

ilya-lavrenov reviewed Sep 20, 2023

View reviewed changes

ilya-lavrenov assigned akashchi Sep 20, 2023

rkazants reviewed Sep 20, 2023

View reviewed changes

.github/workflows/linux.yml Show resolved Hide resolved

ababushk and others added 3 commits September 21, 2023 12:43

Update .github/workflows/linux.yml

f98efa8

Co-authored-by: Ilya Lavrenov <[email protected]>

Update .github/workflows/linux.yml

71b15f1

Co-authored-by: Ilya Lavrenov <[email protected]>

Fix review comments

f05c6ea

ilya-lavrenov reviewed Sep 22, 2023

View reviewed changes

ababushk added 4 commits September 22, 2023 14:43

More review comment fixes

79aa469

Remove DEBIAN_FRONTEND from Python Models job

e726d78

Merge branch 'master' into aks_runners

07e21d8

"Get tools versions" step is absent on master

d1e73f0

ilya-lavrenov approved these changes Sep 22, 2023

View reviewed changes

ilya-lavrenov merged commit 3e82849 into openvinotoolkit:master Sep 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[workflows/linux] Switch to self-hosted runners in Azure Kubernetes Service #19183

[workflows/linux] Switch to self-hosted runners in Azure Kubernetes Service #19183

ababushk commented Aug 14, 2023

ababushk commented Sep 18, 2023

ababushk commented Sep 18, 2023

mryzhov commented Sep 22, 2023

ilya-lavrenov Sep 22, 2023

rkazants Sep 22, 2023

ilya-lavrenov Sep 23, 2023

[workflows/linux] Switch to self-hosted runners in Azure Kubernetes Service #19183

[workflows/linux] Switch to self-hosted runners in Azure Kubernetes Service #19183

Conversation

ababushk commented Aug 14, 2023

Details:

Tickets:

ababushk commented Sep 18, 2023

ababushk commented Sep 18, 2023

mryzhov commented Sep 22, 2023

ilya-lavrenov Sep 22, 2023

Choose a reason for hiding this comment

rkazants Sep 22, 2023

Choose a reason for hiding this comment

ilya-lavrenov Sep 23, 2023

Choose a reason for hiding this comment