Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[v1.8.x] Backport PRs from v1.x branch #19946

Closed
wants to merge 6 commits into from
Closed

[v1.8.x] Backport PRs from v1.x branch #19946

wants to merge 6 commits into from

Commits on Feb 24, 2021

  1. [v1.x] Migrate to use ECR as docker cache instead of dockerhub (apach… 

    …e#19654)
    @josephevans
    josephevans authored and Joe Evans committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    7e38dcb View commit details
    Browse the repository at this point in the history

  2. [v1.x] Update CI build scripts to install python 3.6 from deadsnakes … 

    …repo (apache#19788)

* Install python3.6 from deadsnakes repo, since 3.5 is EOL'd and get-pip.py no longer works with 3.5.

* Set symlink for python3 to point to newly installed 3.6 version.

* Setting symlink or using update-alternatives causes add-apt-repository to fail, so instead just set alias in environment to call the correct python version.

* Setup symlinks in /usr/local/bin, since it comes first in the path.

* Don't use absolute path for python3 executable, just use python3 from path.

Co-authored-by: Joe Evans <[email protected]>
    @josephevans
    josephevans and Joe Evans committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    65ae3a8 View commit details
    Browse the repository at this point in the history

  3. Disable unix-gpu-cu110 pipeline for v1.x build since we now build wit… 

    …h cuda 11.0 in windows pipelines. (apache#19828)

Co-authored-by: Joe Evans <[email protected]>
    @josephevans
    josephevans and Joe Evans committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    399ac7d View commit details
    Browse the repository at this point in the history

  4. [v1.x] For ECR, ensure we sanitize region input from environment vari… 

    …able (apache#19882)

* Set default for cache_intermediate.

* Make sure we sanitize region extracted from registry, since we pass it to os.system.

Co-authored-by: Joe Evans <[email protected]>
    @josephevans
    josephevans and Joe Evans committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    e81d2b9 View commit details
    Browse the repository at this point in the history

  5. [v1.x] Address CI failures with docker timeouts (v2) (apache#19890) 

    * Add random sleep only, since retry attempts are already implemented.

* Reduce random sleep to 2-10 sec.

Co-authored-by: Joe Evans <[email protected]>
    @josephevans
    josephevans and Joe Evans committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    e35eb2e View commit details
    Browse the repository at this point in the history

  6. [v1.x] CI fixes to make more stable and upgradable (apache#19895) 

    * Test moving pipelines from p3 to g4.

* Remove fallback codecov command - the existing (first) command works and the second always fails a few times before finally succeeding (and also doesn't support the -P parameter, which causes an error.)

* Stop using docker python client, since it still doesn't support latest nvidia 'gpus' attribute. Switch to using subprocess calls using list parameter (to avoid shell injections).

See docker/docker-py#2395

* Remove old files.

* Fix comment

* Set default environment variables

* Fix GPU syntax.

* Use subprocess.run and redirect output to stdout, don't run docker in interactive mode.

* Check if codecov works without providing parameters now.

* Send docker stderr to sys.stderr

* Support both nvidia-docker configurations, first try '--gpus all', and if that fails, then try '--runtime nvidia'.

Co-authored-by: Joe Evans <[email protected]>
    @josephevans
    josephevans and Joe Evans committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    fe354b5 View commit details
    Browse the repository at this point in the history