From 2469f06105128272391d3633111d197a250afd35 Mon Sep 17 00:00:00 2001 From: Jack Taylor <108682042+jataylo@users.noreply.github.com> Date: Tue, 11 Apr 2023 07:24:35 +0100 Subject: [PATCH] IFU 2023-03-24 (#29) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Make sure package_type is set (#1139) * Update check_binary.sh * Update check_binary.sh * Modifying smoke test to add more advanced validation as requested (#1124) * Modify smoke test matrix More vision smoke tests Temporary pointing to my repo for testing Try 2 use atalman builder Modify path Fixing commits Testing Testing Smoke test modifications Refactor test code Fix typo Fixing image read A little more refactoring Addressing comments Testing * Add same test for windows and macos * Addressing c omments * Add manywheel special build for including pypi package (#1142) * Add manywheel special build Testing Builder change Testing Adding manywheel cuda workflow Simplify Fix expr * address comments * checking for general setting * Pass correct parameters for macos validations (#1143) * Revert "Update check_binary.sh" This reverts commit 6850bedc88c4f6dd21311758d57b8d39d80efd22. * Revert "Update check_binary.sh" This reverts commit 051b9d1784c8f5461106537527fd66d5d75dc9c4. * setup periodic test to run binary verification  pytorch/pytorch#84764: (#1144) * add a reusable workflow to run all smoke tests/or smoke tests for a specific os/channel * add workflows to schedule the periodic smoke tests for nightly and release channels * Update aarch64 script to latest one (#1146) * minor: fix the typo job name for windows binaries validation workflow (#1147) * fix the typo in the the job name for the release binaries validation workflow (#1148) issue was introduced in #1144 * Move to rc2 of 3.11 python (#1149) Need it to get several convenience functions * Integrates CUDA pip wheels (#1136) * Refactors rpath to externally set var. Adds mechanism to add metadata * Sets RUNPATH when using cudnn and cublas wheels * Escapes dollar sign * Fix rpath for cpu builds Co-authored-by: atalman * Uses RPATH instead of RUNPATH so that user strictly uses pypi libs (#1150) * Binary Validation Workflow - Adding check binary script (#1127) * Update action.yml * Update validate-macos-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Update validate-linux-binaries.yml * Fix check binary for arm64 (#1155) * Fix check binary for arm64 * Update check_binary.sh Co-authored-by: Nikita Shulga Co-authored-by: Nikita Shulga * Fix for including nvtx dll and cudart (#1156) * Fix for invluding nvtx dll and cudart * Fix for include nvtx * Fix spaces * Back out inclusion of cudart (#1157) * Add cuda and date check to smoke test (#1145) * shorten binary validation workflow names, so they are more readable in the HUD and GH job view (#1159) * Fix anaconda torchaudio smoke test (#1161) * Fix anaconda torchaudio smoke test * Format using ufmt * Fix whels tests for torchaudio (#1162) * Pin condaforge version Most recent version fails with invalid cert error when trying to update python * Option to run resnet classifier on specific device * Fix typo `.test/smoke_test` -> `test/smoke_test` Noticed when pushed https://github.com/pytorch/builder/commit/3b93537e29059042dd4c4d96c5ee7a72f0fd0500 and no tests were run * Test resnet classifier on CUDA (#1163) * [ROCm] support for rocm5.3 wheel builds (#1160) * Updates to support rocm5.3 wheel builds (#6) * Changes to support ROCm 5.3 * Updated as per comments * Installing python before magma build - In ROCm 5.3 libtorch build are failing during magma build due to to missing python binary so added install statement * Move python install to libtorch/Dockerfile (#8) * Updating the condition for noRCCL build (#9) * Updating the condition for noRCCL build * Updated changes as per comments * Use MIOpen branch for ROCm5.3; Change all conditions to -eq * Use staging branch of MIOpen for ROCm5.3 * Fix merge conflict Fix merge conflict Co-authored-by: Pruthvi Madugundu Co-authored-by: Pruthvi Madugundu Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com> Co-authored-by: Jithun Nair * Validate python 3.11 (#1165) * Validate python 3.11 * Validate linux binaries change Add options Import torchvision Adding python 3.11 install pass package to check nightly binaries date Test test Add python 3.11 code testing Adding python 3.11 test Add python 3.11 validation Adding zlib develop install Install zlib etc.. Adding zlib1g as well testing testing Adding validate windows binary Trying to workaround testing Refacor smoke test Add import statement fix datetime call * Fix stripping dev * fix import * Strip pypi-cudnn from the version.py (#1167) * Strip pypi-cudnn from the version.py * small fix * Regenerates RECORD file to reflect hash changes caused by sed'ing the version suffix (#1164) * Add pypi cudnn package to tests (#1168) * Add pypi cudnn package to tests * Fix pypi installation check * Fix pypi instructions setting * Update DEVELOPER_DIR in build_pytorch.sh Not sure why we are still expecting Xcode9 to be present there, update it to the same folder as wheel builds May be fixes https://github.com/pytorch/pytorch/issues/87637 * Fix to not use sccache if it's not setup properly (#1171) * Revert "Fix to not use sccache if it's not setup properly (#1171)" (#1172) This reverts commit 377efea5040e585708401e671f767972f183aa2e. * Remove cuda102 and cuda115 docker builds and regenerate manylinux docker (#1173) * Rebuild manywheel * Remove cuda102 and cuda115 * [aarch64] add mkldnn acl backend build support for pytorch cpu libary (#1104) * Only push to Docker and Anaconda repo from main (#1175) We currently allow push from any branch to go to Docker (and Anaconda) prod. This is a dangerous practice because it allows unfinished works to jump to prod and used by other workflows * Release 1.13 script changes (#1177) * Test ResNet on MPS (#1176) After https://github.com/pytorch/pytorch/issues/86954 is fixed, we should be able to test resnet on MPS * Revert "Test ResNet on MPS (#1176)" (#1180) This reverts commit efa1bc745bfc1dd4a2adc92fb4fdfe19dd745d4c. * Add v1.13 versions * Update CMake to 3.18, needed for C++17 compilation (#1178) * release: separate out version suffixes for torch pypi promotion (#1179) * Fixup wheel published to PyPI (#1181) * Fixup wheel published to PyPI * Update prep_binary_for_pypi.sh * Fix folder deletion for pypi prep Co-authored-by: Andrey Talman * Update cmake version to 3.18 for libtorch docker * Pins cuda runtime to 111.7.99 (#1182) * Fixes cuda pypi rpaths and libnvrtc name (#1183) * Allow ROCm minor releases to use the same MIOpen branch as the major release (#1170) * Allow ROCm minor releases to use the same MIOpen branch as the major release * correct logic to ensure rocm5.4 doesn't fall in wrong condition * add 11.8 workflow for docker image build (#1186) * Using windows runners from test-infra for validation workflows (#1188) * Testing new windows runners test Testing Testing testing testing test Test Testing testing Testing Testing test Test test testing testing Test testing test testing testing testing testing testing testing test test testing testing testing testing Test test test testing testing testing testing testing testing testing testing testing Refactor code * Adding details for the test-infra issue * Update current CUDA supported matrix * add magma build for CUDA11.8 (#1189) * Test setting job name (#1191) * Use official Python-3.11 tag (#1195) * remove CUDA 10.2-11.5 builds (#1194) * remove CUDA 10.2-11.5 builds * remove 11.5 and 11.3 builds * build libtorch and manywheel for 11.8 (#1190) * build libtorch and manywheel for 11.8 * Update common/install_magma.sh * use magma-cuda build-1 by default; remove CUDA 10.2-11.5 builds Co-authored-by: Andrey Talman * [Validation] Pass ref:main to general worker (#1197) * Pass ref:main to general worker * Try to pass reference to workflow * Pass ref:main to general worker * Test * Pass reference as input parameter * Make new variable not required * Fix typo * Add workflow for manywheel cpu-cxx11-abi (#1198) * [Validation] Use linux_job for linux workers (#1199) * Use linux_job for linux workers Test Testing Test testing Tetsing testing Change linux binary action test Simplify version check * Fix if statement * Fix typo * Fix cuda version check Fix Audio and Vision version check Add check binary to libtorch test test testing testing testing Testing Testing testing * Use macos generic workers (#1201) * Use macos generic workers fix workflow testing Add arm64 builds test Remove validate binary action * add check binary step * fix ld_library path * add package type * Adding ref to validate binaries (#1204) * ROCm5.3 nightly wheels (#1193) * Enable ROCm5.3 nightly wheels * Enable ROCm5.3 docker builds * Update amdgpu repo url for ROCm5.3 * ROCm5.3 not supported on Ubuntu 18.04 * empty * Another empty commit * Try disabling MLIR build to shorten docker build time * Clean up disk space * MLIR project changed names from ROCm5.4 * Retrigger CI to get around flaky magma git access error * One more cmake-3.18.4 update * Use cmake-3.18 for ROCM builds * More cmake ROCM tweaks * cmake-3.18 installation on ROCM (take 3) * add conda builds for CUDA 11.8 (#1205) * Enable nightly CUDA 11.8 builds (#1206) * enable nightly builds for CUDA 11.8 * add CUDA 11.8 version to manywheel, remove 11.3 and 11.5 * Windows CUDA 11.8 changes (#1207) * Add continue on error to validation jobs (#1209) * Add continue on error to validation jobs * test * Delete unmaintaned torchvision build scripts (#1210) All build logic has long moved to torchvision repo and now is executed by reusable workflow from https://github.com/pytorch/test-infra/tree/main/.github/workflows * build_pytorch.sh replace tabs with spaces (#1211) * Make PyTorch depend on TorchTrition (#1213) Remove me when Triton is properly released elsewhere * Remove smoke test script that is no longer used (#1212) * Another tabs-to-spaces change `s/\t/\ \ \ \ \ \ \ \ /` * Disable continue on error (#1214) * Add torchtrition dependency for wheels * Make PyTorchConda depend on Triton (Take 2) Multi-line environment variables are hard, so lets do it traditional way * Revert "Add torchtrition dependency for wheels" This reverts commit 475100b613a11bf68f9fd5c75e8054022449ad36. * Add TorchTrition dependency for wheels (take 2) Now tests should be green thanks to https://github.com/pytorch/pytorch/pull/90017 * Add sympy to pytorch linux dependencies * Mitigate windows nightly build regressions By pinning conda to 22.9.0 Fixes https://github.com/pytorch/pytorch/issues/90059 * Consolidating validation scripts (#1219) * Consolidating validation scripts * Fix validate script name * Correct script path * Correct script path * test * testing * testing * testing * testing * test * test * test * testing * testc * test hook * adding wondows use case * windows use case * test * testing * Windows fixes * more fixes * Add package type * testing more * Truncate RECORD instead of delete (#1215) * Refactor and fix windows smoke tests (#1218) * Fix windows smoke test * Fix first if statement * Refactor not to cal install nightly package * Revert "Refactor not to cal install nightly package" This reverts commit ac580c8cce2db1d986b256a66322cb9c3a2fd918. * Fix pip install command remove cu102 * Refacor the conda installation * Add cuda profiler apu to cuda install 11.8 (#1221) * Update CUDA upgrade runbook to mention subpackages changes As per following doc: https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html * conda: Add CUDA_HOME, cuda binaries to path (#1224) * Refactor macos-arm64 into separate group (#1226) * Adding libcufft constraint (#1227) * Adding libcufft constraint * Adding rest of the dependencies * Advance build number in pytorch-cuda (#1229) * Make sympy mandatory dependency of PyTorch Should fix https://github.com/pytorch/audio/actions/runs/3684598046/jobs/6234531675 * Revert me later: Fix conda package smoke tests * Install `sympy` via pip rather than conda Needs to be reverted as well * Refactor smoke tests to configure module included in the release (#1223) * Changes to prep for pypi script for release 1.13.1 (#1231) * PyPi binary validation and size check (#1230) * Validate binary size * Validate binary size linux_job * evaluate the fix from #1231 * Add an optional artifact upload, consolidate fixes to `prep_binary_for_pypi.sh` * Adding new workflow to call from domain libraries to validate on domain libraries such as text (#1234) * Testing new workflow Fix naming fix input * Changed comments * Ad ability to call validate domain library manually (#1235) * Adding test for validate dm workflow and fixing dm validation workflow (#1236) * Test manywheel packages (#1239) Change only docker file * Bump scripts in release (#1241) * release: Strip whitespace from version_with_suffix (#1242) * Cuda 11.8 and removal of dev packages (#1243) * Adding more OS's to validate domain library workflow (#1238) * Adding more OS's to validate domain library workflow * conda and wheel togeather * add macos workflows * fix workflow * Add target os variable to windows validation (#1244) * Update MKL to 2022.1 (#1245) As previous one occasionally crashes on AMD CPUs May be addresses https://github.com/pytorch/pytorch/issues/89817 Please note, that in order to get maximum perf on AMD CPUs one needs to compile and LD_PRELOAD following library: ``` int mkl_serv_intel_cpu_true() { return 1; } ``` * Adds infra to use nvidia dependencies from pypi and cleans up patches (#1196) * Installs NCCL from redist, uses system NCCL, and adds pypi RPATH * Cleans up nvrtc patches and adds it using main script * Fixes typo * Adds more dependencies and builds torch with dynamic linking * NCCL dirs have to be specified. Otherwise picks up different version * Handles 11.8 * Adds echo message for nccl 2.15 * Remove invalid git option (#1246) * Revert "Adds infra to use nvidia dependencies from pypi and cleans up patches (#1196)" (#1247) This reverts commit ee59264c887b08514463894460e3803892ae5c3c. * Add with_cuda flag (#1249) * Add GPU architecture env variables (#1250) * Add cuda to jobname for validate domain library (#1252) * Remove pylief dependency (#1255) * Fix PEP503 for packages with dashes * Rename `torchtriton` to `pytorch-triton` Companion change for https://github.com/pytorch/pytorch/pull/91539 * s3_management: Hide specific packages between dates (#1256) * s3_management: Pin requirements.txt Packaging got updated and that's not what we want Signed-off-by: Eli Uriegas * s3_management: except ValueError Signed-off-by: Eli Uriegas * s3_management: Use the correct format for strptime Signed-off-by: Eli Uriegas * s3_management: Bump bad dates to october 17th (#1257) * s3_management: hide torchtriton (#1258) * s3_management: Add PACKAGE_ALLOW_LIST for indices (#1259) * s3_management: Bump bad date end to 12/30 (#1260) * Adds infra to use nvidia dependencies from pypi and cleans up patches (#1248) * Installs NCCL from redist, uses system NCCL, and adds pypi RPATH * Cleans up nvrtc patches and adds it using main script * Fixes typo * Adds more dependencies and builds torch with dynamic linking * NCCL dirs have to be specified. Otherwise picks up different version * Handles 11.8 * Adds echo message for nccl 2.15 * Fixes logic for 11.8 and adds missing names for DEPS_SONAME * s3_management: Account for underscore packages pytorch-triton is listed as pytorch_triton Signed-off-by: Eli Uriegas * s3_management: simplify allowlist, correct underscores Signed-off-by: Eli Uriegas * Fix cuda version in nightly (#1261) * Adding py311 validations (#1262) * Use MATRIX_* variables instead of redeefining new var each time (#1265) * Fix validation domain library (#1266) remove ref main fix workflow more refactor * Nightly: do test install with the dependencies better and skip CUDA tests on cpu only box (#1264) * Refactor PyTorch wheel and libtorch build scripts for ROCm (#1232) * Refactor wheel and libtorch build scripts (https://github.com/ROCmSoftwarePlatform/builder/pull/7) * Update to so patching for ROCm Wildcard used in grep to grab the actual numbered so file referenced in patchelf. This allows the removal of specifying the so number in DEPS_LIST & DEPS_SONAME This commit also adds the functionality for trimming so names to build_libtorch.sh from build_common.sh * Refactor to remove switch statement in build_rocm.sh This commit refactors build_rocm.sh and brings in a few major updates: - No longer required to specify the full .so name (with number) for ROCm libraries - The .so versions are copied and the patching code will fix the links to point to this version - No longer required to specify paths for ROCm libraries allowing the removal of the large switch - Paths are acquired programmatically with find - No longer required to specify both the path and filename for the OS specific libraries - Programatically extract file name from the path - Automatically extract Tensile/Kernels files for the architectures specified in PYTORCH_ROCM_ARCH and any non-arch specific files e.g. TensileLibrary.dat * rocfft/hipfft link to libhiprtc.so in ROCm5.4 (https://github.com/ROCmSoftwarePlatform/builder/pull/15) Co-authored-by: Jack Taylor <108682042+jataylo@users.noreply.github.com> * add sm_90 to CUDA11.8 builds (#1263) * add sm_90 to CUDA11.8 builds * Manually invoke bash for Miniconda * Revert "add sm_90 to CUDA11.8 builds (#1263)" (#1275) This reverts commit e1453a4913027020519b2cca9da18476a6aded17. * Set ubuntu distribution correctly for ROCm5.3 and above (#1268) * Fix unbound variable error (#1276) Regression introduced (and ignored) by https://github.com/pytorch/builder/pull/1262 Test plan: ``` % bash -c 'set -u; if [[ -z "${FOO}" ]]; then echo "bar"; fi' bash: FOO: unbound variable (base) nshulga@nshulga-mbp builder % bash -c 'set -u; if [[ -z "${FOO+x}" ]]; then echo "bar"; fi' bar (base) nshulga@nshulga-mbp builder % FOO=1 bash -c 'set -u; if [[ -z "${FOO+x}" ]]; then echo "bar"; fi' ``` * Manually invoke bash for miniconda (#1277) Fixes build issues failing with: ``` ./Miniconda3-latest-Linux-x86_64.sh: 438: ./Miniconda3-latest-Linux-x86_64.sh: [[: not found ``` as seen in e.g.: https://github.com/pytorch/builder/pull/1271 * Fix perm Which somehow got changed by https://github.com/pytorch/builder/commit/62103bfac910e64b4554f17b7968f9aef0dbbcb3 * add sm_90 to CUDA11.8 builds (#1278) * libtinfo.so version update and logic fix for ROCm libtorch (#1270) * Use libtinfo.so.6 for Ubuntu 2004 * Fix to origname grep * Condition on ROCM_VERSION for libtinfo6 * Looks like it is not used anywhere. (#1273) * Build Windows binaries with Visual Studio 2022 Build Tools (#1240) * Build Windows binaries with Visual Studio 2022 Build Tools * Unify casing in Batch files, remove VS 2017 installation * Remove VS 2017 Conda scripts, unify casing in conda Batch scripts, minor Conda scripts tweaks * Slim down `pytorch-cuda` It should only contain runtime dependencies that PyTorch+domain libraries depend on, namely: - cudart - cublas - cusparse - cufft - curand - nvtx - nvrtc - nvjpeg (for TorchVision) This removes dependencies on NVCC, build/debug tools, etc which are not needed for running the pytorch Test Plan: `conda create -n tmp -c nvidia -c malfet cuda-toolkit==11.7` and observe that only relevant packages are installed Fixes https://github.com/pytorch/pytorch/issues/91334 * [BE] Delete `unicode-flags` build options (#1284) There were relevant only for Python<=3.3 * [BE] Define `openssl_flags` (#1285) Rather than have two invocations of `./configure` * Build with `--enabled-shared` if `patchelf` is found (#1283) This is needed to make `manylinux-wheel` images usable for building new Triton binaries. Test plan: Build docker and verify that following `CMakeLists.txt` finishes successfully: ``` cmake_minimum_required(VERSION 3.6) find_package(Python3 REQUIRED COMPONENTS Interpreter Development) message(WARNING Executable ${Python3_EXECUTABLE}) message(WARNING IncludeDirs ${Python3_INCLUDE_DIRS}) message(WARNING Libraries ${Python3_LIBRARIES}) ``` * Update cudnn to 8.7.0.84 for CUDA 11.8 builds (#1271) * update cudnn to 8.7.0.84 for CUDA 11.8 builds * workaround for #1272 * Revert "workaround for #1272" This reverts commit c0b10d805ad7ea83f00e8841bf3d4ad6404fbaa4. * update cudnn==8.7.0.84 for windows * [BE] Remove references to Python<3.6 (#1287) * Upgrade desired python versoin to 3.8 For libtorch builds * Fix how libtorch picks the python version * Tweak conda builds to support 3.11 Add `-c malfet` when building for 3.11 (though perhaps it's better to move numpy to pytorch channel) Tweak some build time dependencies * Fix typo * Skip triton dependency for 3.11 CUDA builds * Update build-number to 3 * Add ability to override cuda archs for conda (#1282) * [ROCm] reduce disk space used in image (#1288) Fixes https://github.com/pytorch/builder/issues/1286 * Extend MacOS/Windows builds to 3.11 By installing dependencies from pip Should be a no-op for <=3.10 * ci: Migrate to checkout@v3 (#1290) checkout@v2 is deprecated moving to checkout@v3 Signed-off-by: Eli Uriegas * Fix typo * Add 3.11 option for Windows builds * Add python-3.11 download location for windows * Add pypi with cudnn package test (#1289) * Add pypi with cudnn package test * Add pypi with cudnn package test * test * test * More pypi cudnn changes * test * Fix pipy smoke test * Remove debug comments * Delete some ancient checks for MacOS builds As we no longer build for Python-2.7 or 3.5 * Add libnvjpeg-dev package as fallback (#1294) * Add libnvjpeg-dev package as fallback * Move libnvjpeg and libnvjpeg-dev to required packages * Update conda/pytorch-cuda/meta.yaml --------- Co-authored-by: Nikita Shulga * Upgrade nightly wheels to rocm5.4.2 (#1225) * Upgrade nightly wheels to rocm5.4 * Adding graphic architectures for ROCm 5.4 * Updated to use ROCm5.4.1 * Updated to use ROCm5.4.2 * Fixed syntax error * Perform build on image with magma and miopen preinstalled * Add dev packages for windows pytorch-cuda dependencies (#1295) * Add dev packages for windows dependencies * Adding architecture dependent builds * Add notes around windows * fix typo * Bumping version to v3 * rocm libtorch prebuild magma; fix manylinux cmake version (#1296) * Add manywheel:cpu-cxx11-abi checkup for check_binary.sh (#1251) * Remove with_py311 flag (#1301) * rocm manylinux now uses devtoolset 9 (#1300) * fix ACL_ROOT_DIR setting and upgrade the ACL version to 22.11 (#1291) * Add `-c malfet` for Windows builds as well * Set torch._C._PYBIND11_BUILD_ABI version check only for GLIBCXX_USE_CXX11_ABI=0 (#1303) * Adding limit windows builds logic (#1297) * Adding limit windows builds logic * Remove empty space * Simplify mkl build dependencies (#1305) On Linux and Mac PyTorch must be built against `mkl=2020.x` in order to be compatible with both `mkl-2021` and `mkl-2022`, that added `.so.1` and `.so.2` files respectively, that would make binary linked against those versions incompatible with the newer/older toolchains. This is not an issue on Windows, as all mkl binaries there end with simple `.dll` * "Fix" PyTorch CPU conda testing It's still horribly broken, but make it a bit better by not installing pytorch from default anaconda channel (which installs 1.12.1 that does not have any dependencies 2.0 dev package supposed to have) For example, see this runlog https://github.com/pytorch/pytorch/actions/runs/4155371267/jobs/7189101147 * Update torch._C._PYBIND11_BUILD_ABI version check (#1306) * Skip tests for manywheel built with _GLIBCXX_USE_CXX11_ABI=1 * Put back smoke test label (#1310) * [aarch64] add support for torchdata wheel building (#1309) * Python 3.11 validation workflow tests (#1304) * Test windows py311 * Nightly binaries * Fix py311 tests * fix python calling * Revert "Nightly binaries" This reverts commit cbf80ca5fdb8a2fce5552591ebe0f60dd87d04fe. * add a scheduled workflow for the nightly pypi binary size validation (compliments pytorch/test-infra#2681) (#1312) * Add regression test for https://github.com/pytorch/pytorch/issues/94751 * Add 3.11 and `--pytorch-only` options * Add `lit` to list of allowed packages As it is now mandatory (albeit spurious) dependency of pytorch-triton See https://pypi.org/project/lit/ for more details * s3: Allow tar.gz as an accepted file extension (#1317) * Changes for Python 3.11 and smoke Test RC cut (#1316) * Smoke Test RC cut * Validate binaries 3.11 * test * Smoke test binaries * Fix pytorch-cuda chan download * Remove temp change * Make sure we don't use GPU runners for any of libtorch validations (#1319) * Make sure we don't use GPU runners for any of libtorch * Make sure we don't use GPU runners for any of libtorch * s3: Add pytorch_triton_rocm to index (#1323) Signed-off-by: Eli Uriegas * s3: Add tqdm package req for text (#1324) * Add `--analyze-stacks` option That using `git rev-base`, prints total number of stacks, and its average, mean and max depth At the time of submission here is top 10 ghstack uses of pytorch: ``` ezyang has 462 stacks max depth is 15 avg depth is 1.70 mean is 1 awgu has 240 stacks max depth is 28 avg depth is 4.30 mean is 1 peterbell10 has 146 stacks max depth is 7 avg depth is 1.84 mean is 1 zou3519 has 128 stacks max depth is 7 avg depth is 1.98 mean is 1 jerryzh168 has 113 stacks max depth is 16 avg depth is 1.45 mean is 1 bdhirsh has 111 stacks max depth is 7 avg depth is 1.85 mean is 2 wconstab has 108 stacks max depth is 7 avg depth is 2.15 mean is 1 SherlockNoMad has 99 stacks max depth is 4 avg depth is 1.24 mean is 1 zasdfgbnm has 80 stacks max depth is 11 avg depth is 2.52 mean is 6 desertfire has 73 stacks max depth is 3 avg depth is 1.14 mean is 1 ``` * Add filelock and networkx deps (#1327) To match dependencies for wheel files defined in https://github.com/pytorch/pytorch/blob/ed1957dc1989417cb978d3070a4e3d20520674b4/setup.py#L1021-L1024 * Remove building magma from source * Revert * Upgrade cmake version to 3.22.1 to build triton (#1331) * Upgrade cmake version to 3.22.1 to build triton * Pin patcheft version * Fix comment typo * Smoke test for cuda runtime errors (#1315) * Add test for cuda runtime errors * Add cuda exception smoke test * Move cuda runtime error to end * Move cuda runtime error to end * Address comments * Address comments * Add Jinja2 Dependency (#1332) As part of the effort to fix https://github.com/pytorch/pytorch/issues/95986 * Add MarkupSafe to S3 Index (#1335) * Remove rocm5.1 rocm5.2 from libtorch Dockerfile * [aarch64] Adding CI Scripts to build aarch64 wheels (#1302) * add aarch64 ci scripts * added readme. get branch from /pytorch * Add smoke tests conv,linalg,compile. And better version check. (#1333) * Add smoke tests conv,linalg,compile * Add version check * Fix typo Fix version check Add not * Add exception for python 3.11 * fix typo * Try to exit after CUDA Runtime exception * Restrict carsh test only to conda * Restrict carsh test only to conda * Fix tests * Turn off cuda runtime issue * tests * more tests * test * remove compile step * test * disable some of the tests * testing * Remove extra index url * test * Fix tests * Additional smoke tests Remove release blocking changes * Aarch64 changes for PyTorch release 2.0 (#1336) * Aarch64 changes for PyTorch release 2.0 * Fix spacing * Update aarch64_linux/build_aarch64_wheel.py Co-authored-by: Nikita Shulga * Update aarch64_linux/build_aarch64_wheel.py Co-authored-by: Nikita Shulga --------- Co-authored-by: Nikita Shulga * Aarch64 build py3.11 fix (#1341) * Fix nightly smoke test (#1340) * Fix nightly smoke test * Fix nightly builds * Release 2.0 release scripts changes (#1342) * Release 2.0 release scripts changes * Release script modifications * Add more packages to allow list (#1344) * Add `jinja2` dependency to conda package To be consistent with wheels, see https://github.com/pytorch/pytorch/95961 * Restrict jinja to py 3.10 or less (#1345) * Update `torchtriton` version to 2.1.0 * And update trition version here as well * added smoke test for max-autotune (#1349) Co-authored-by: agunapal * Refactor conda backup script (#1350) * Refacto conda backup * Fix space * Minor style * Revert "Upgrade cmake version to 3.22.1 to build triton (#1331)" (#1351) * Revert "Upgrade cmake version to 3.22.1 to build triton (#1331)" This reverts commit 18c5017d9eedd2e5e203608e9ad6c05a2324b3ac. * Selective revert * Get cmake from pip * Use 3.18.2 from conda * Release script changes, add more release dependencies, bump version for aarch64 builds (#1352) * Release script changes * Add Jinja2 dependency * Fix typo * Add pytorch conda dependencies (#1353) * Add latest dependencies for pytorch 2.0 release (#1357) * Fix typo * Revert "Revert me later: Fix conda package smoke tests" This reverts commit d7f2a7c0d8d80a9efe4d2b245a292de8f2528b8c. * [aarch64] update readme with the "--enable-mkldnn" option (#1362) This needs to be enabled for official wheel building. * Replace `--enable-mkldnn` with `--disable-mkldnn` Also, change default to ubuntu-20.04 * Update AMIs Using following images: ``` % aws ec2 describe-images --image-ids ami-078eece1d8119409f ami-052eac90edaa9d08f ami-0c6c29c5125214c77 --query "Images[].[ImageId, Description]" [ [ "ami-078eece1d8119409f", "Canonical, Ubuntu, 18.04 LTS, arm64 bionic image build on 2023-03-02" ], [ "ami-0c6c29c5125214c77", "Canonical, Ubuntu, 22.04 LTS, arm64 jammy image build on 2023-03-03" ], [ "ami-052eac90edaa9d08f", "Canonical, Ubuntu, 20.04 LTS, arm64 focal image build on 2023-03-01" ] ] ``` * Update tags for domain libraries * Add PyTorch version pinning to release wheels * Fix flake8 * [BE] Introduce `build_domains` function And call it to rebuild only domains if torch wheel is available * Switch deprecated ubuntu-18.04 runner to ubuntu-latest (#1334) * Switch deprecated ubuntu-18.04 runner to self-hosted 2xlarge * Leave build-nvidia-docker for now * Apply suggestions from code review Co-authored-by: Nikita Shulga * Use ephemeral runners * Use ubuntu-latest * Apply suggestions from code review Co-authored-by: Nikita Shulga * Switch from latest to 22.04 to pin the version --------- Co-authored-by: Nikita Shulga * Introduce optional --build-number parameter * Revert me later: Fix conda package smoke tests (cherry picked from commit d7f2a7c0d8d80a9efe4d2b245a292de8f2528b8c) Alas, it's still used and causes nightly build failures * Fix aarch64 torchvision build (#1363) * Fix torchvision image extension compilation * Fix torchvision image extension compilation * Set enable_mkldnn to pypi build * Remove unused `enable_mkldnn` for configure_system * [aarch64] Try to link statically with png/jpeg Also, add testing (which is currently broken) * Revert "Revert me later: Fix conda package smoke tests" This reverts commit ce427de8a88675c584f63008f4bb9164ab9eef79. * [AARCH64] Fix image.so wheel By adding explicit libz dependency * [AARCH64] Pass `BUILD_S3` to torchdata To make build consistent with Linux-x86_64 * Revert "[AARCH64] Pass `BUILD_S3` to torchdata" This reverts commit ae8e825450eeee909a354433625df63027a0d155. As it does not want to be built on aarch64 * Add portalocker (#1364) * [BE] Error handling in build_aarch64_wheel I've noticed that build errors in `build_ArmComputeLibrary` would be ignored as semicolon is used between the commands, instead of && Also, replace nightly version evaluation by relying on torch, to rely on individual libraries * [AArch64] Pass `args.instance_type` to `start_instance` * use c++17 when building windows smoke tests (#1365) Summary: We are seeing failures during CI dealing with some headers that have nested namespaces. This is expected to remedy them. One such example: https://github.com/pytorch/pytorch/actions/runs/4510336715/jobs/7942660912 Test Plan: Test this with CI. --------- Signed-off-by: Eli Uriegas Signed-off-by: Eli Uriegas Co-authored-by: Andrey Talman Co-authored-by: andysamfb <111015134+andysamfb@users.noreply.github.com> Co-authored-by: izaitsevfb <108101595+izaitsevfb@users.noreply.github.com> Co-authored-by: Nikita Shulga Co-authored-by: Syed Tousif Ahmed Co-authored-by: Syed Tousif Ahmed Co-authored-by: Nikita Shulga Co-authored-by: Wei Wang <109318740+weiwangmeta@users.noreply.github.com> Co-authored-by: Nikita Shulga Co-authored-by: Pruthvi Madugundu Co-authored-by: Pruthvi Madugundu Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com> Co-authored-by: Jithun Nair Co-authored-by: Huy Do Co-authored-by: snadampal <87143774+snadampal@users.noreply.github.com> Co-authored-by: Eli Uriegas <1700823+seemethere@users.noreply.github.com> Co-authored-by: ptrblck Co-authored-by: zhuhong61 <95205772+zhuhong61@users.noreply.github.com> Co-authored-by: Greg Roodt Co-authored-by: Eli Uriegas Co-authored-by: Dmytro Dzhulgakov Co-authored-by: albanD Co-authored-by: Radek Bartoň Co-authored-by: divchenko Co-authored-by: Jeff Daily Co-authored-by: Bo Li <110066325+BLOrange-AMD@users.noreply.github.com> Co-authored-by: Mike Schneider <104035434+xncqr@users.noreply.github.com> Co-authored-by: Ankith Gunapal Co-authored-by: agunapal Co-authored-by: dagitses --- .circleci/scripts/binary_checkout.sh | 4 +- .github/actions/validate-binary/action.yml | 57 ---- .../validate-windows-binary/action.yml | 46 --- .github/scripts/validate_binaries.sh | 50 +++ .github/workflows/build-conda-images.yml | 8 +- .github/workflows/build-libtorch-images.yml | 18 +- .github/workflows/build-llvm-images.yml | 4 +- .github/workflows/build-magma-linux.yml | 6 +- .github/workflows/build-magma-windows.yml | 8 +- .github/workflows/build-manywheel-images.yml | 35 +- .../test-validate-domain-library.yml | 19 ++ .github/workflows/validate-binaries.yml | 90 +++++ .github/workflows/validate-domain-library.yml | 153 +++++++++ .github/workflows/validate-linux-binaries.yml | 119 +++---- .../validate-macos-arm64-binaries.yml | 56 ++++ .github/workflows/validate-macos-binaries.yml | 136 +++----- .../workflows/validate-nightly-binaries.yml | 35 ++ ...alidate-nightly-pypi-wheel-binary-size.yml | 26 ++ .../workflows/validate-release-binaries.yml | 27 ++ .../validate-repackaged-binary-sizes.yml | 88 +++++ .../workflows/validate-windows-binaries.yml | 133 ++++---- CUDA_UPGRADE_GUIDE.MD | 16 +- README.md | 4 + aarch64_linux/README.md | 19 ++ aarch64_linux/aarch64_ci_build.sh | 52 +++ aarch64_linux/aarch64_wheel_ci_build.py | 309 ++++++++++++++++++ .../build_aarch64_wheel.py | 217 ++++++++---- aarch64_linux/embed_library.py | 72 ++++ analytics/github_analyze.py | 40 ++- analytics/validate_binaries.py | 7 +- check_binary.sh | 26 +- common/install_conda.sh | 7 +- common/install_cpython.sh | 48 +-- common/install_cuda.sh | 231 ++++--------- common/install_magma.sh | 15 +- common/install_miopen.sh | 42 ++- common/install_mkl.sh | 4 +- common/install_patchelf.sh | 4 +- common/install_rocm.sh | 4 + common/install_rocm_magma.sh | 5 +- conda/Dockerfile | 24 +- conda/build_all_docker.sh | 2 +- conda/build_docker.sh | 5 + conda/build_pytorch.sh | 57 ++-- conda/build_vision.sh | 94 ------ conda/debugging_pytorch.sh | 2 +- conda/pytorch-cuda/conda_build_config.yaml | 4 + conda/pytorch-cuda/meta.yaml | 73 +++-- conda/pytorch-nightly/bld.bat | 15 +- conda/pytorch-nightly/build.sh | 37 +-- conda/pytorch-nightly/meta.yaml | 17 +- conda/torchvision/bld.bat | 24 -- conda/torchvision/meta.yaml | 53 --- conda/vs2017/conda_build_config.yaml | 24 -- conda/{vs2017 => vs2022}/activate.bat | 35 +- .../conda_build_config.yaml | 9 +- conda/{vs2017 => vs2022}/install_activate.bat | 4 +- conda/{vs2017 => vs2022}/install_runtime.bat | 8 +- conda/{vs2017 => vs2022}/meta.yaml | 10 +- cron/nightly_defaults.sh | 4 +- libtorch/Dockerfile | 36 +- libtorch/build.sh | 2 +- libtorch/build_all_docker.sh | 4 +- libtorch/build_docker.sh | 2 +- magma/Makefile | 39 +-- manywheel/Dockerfile | 42 +-- manywheel/Dockerfile_cxx11-abi | 1 - manywheel/build_all_docker.sh | 4 +- manywheel/build_common.sh | 12 +- manywheel/build_cuda.sh | 224 ++++++------- manywheel/build_docker.sh | 2 +- manywheel/build_libtorch.sh | 8 +- release/promote.sh | 41 ++- release/pypi/prep_binary_for_pypi.sh | 44 ++- release/pypi/promote_pypi_to_staging.sh | 13 +- release/release_versions.sh | 9 +- run_tests.sh | 39 +-- s3_management/backup_conda.py | 22 +- s3_management/manage.py | 86 ++++- s3_management/requirements.txt | 4 +- smoke_test.sh | 197 ----------- test/smoke_test/assets/dog2.jpg | Bin 0 -> 90796 bytes test/smoke_test/assets/rgb_pytorch.jpg | Bin 0 -> 2126 bytes test/smoke_test/assets/rgb_pytorch.png | Bin 0 -> 575 bytes test/smoke_test/smoke_test.py | 268 ++++++++++++--- wheel/build_wheel.sh | 26 +- windows/build_all.bat | 4 +- windows/condaenv.bat | 3 +- windows/cuda102.bat | 58 ---- windows/cuda115.bat | 58 ---- windows/{cuda113.bat => cuda118.bat} | 16 +- windows/internal/check_deps.bat | 16 +- windows/internal/cuda_install.bat | 80 +---- windows/internal/env_fix.bat | 17 +- windows/internal/install_nightly_package.bat | 67 ---- windows/internal/smoke_test.bat | 81 ++--- windows/internal/vc_install_helper.bat | 9 +- windows/internal/vs2017_install.ps1 | 28 -- windows/internal/vs2022_install.ps1 | 56 ++++ windows/internal/vs_install.bat | 8 +- 100 files changed, 2501 insertions(+), 1866 deletions(-) delete mode 100644 .github/actions/validate-binary/action.yml delete mode 100644 .github/actions/validate-windows-binary/action.yml create mode 100755 .github/scripts/validate_binaries.sh create mode 100644 .github/workflows/test-validate-domain-library.yml create mode 100644 .github/workflows/validate-binaries.yml create mode 100644 .github/workflows/validate-domain-library.yml create mode 100644 .github/workflows/validate-macos-arm64-binaries.yml create mode 100644 .github/workflows/validate-nightly-binaries.yml create mode 100644 .github/workflows/validate-nightly-pypi-wheel-binary-size.yml create mode 100644 .github/workflows/validate-release-binaries.yml create mode 100644 .github/workflows/validate-repackaged-binary-sizes.yml create mode 100644 aarch64_linux/README.md create mode 100644 aarch64_linux/aarch64_ci_build.sh create mode 100755 aarch64_linux/aarch64_wheel_ci_build.py rename build_aarch64_wheel.py => aarch64_linux/build_aarch64_wheel.py (74%) create mode 100644 aarch64_linux/embed_library.py delete mode 100755 conda/build_vision.sh delete mode 100644 conda/torchvision/bld.bat delete mode 100644 conda/torchvision/meta.yaml delete mode 100755 conda/vs2017/conda_build_config.yaml rename conda/{vs2017 => vs2022}/activate.bat (57%) mode change 100755 => 100644 rename conda/{torchvision => vs2022}/conda_build_config.yaml (86%) rename conda/{vs2017 => vs2022}/install_activate.bat (98%) mode change 100755 => 100644 rename conda/{vs2017 => vs2022}/install_runtime.bat (92%) mode change 100755 => 100644 rename conda/{vs2017 => vs2022}/meta.yaml (66%) mode change 100755 => 100644 mode change 100644 => 100755 release/pypi/prep_binary_for_pypi.sh delete mode 100755 smoke_test.sh create mode 100644 test/smoke_test/assets/dog2.jpg create mode 100644 test/smoke_test/assets/rgb_pytorch.jpg create mode 100644 test/smoke_test/assets/rgb_pytorch.png delete mode 100644 windows/cuda102.bat delete mode 100644 windows/cuda115.bat rename windows/{cuda113.bat => cuda118.bat} (81%) delete mode 100644 windows/internal/install_nightly_package.bat delete mode 100644 windows/internal/vs2017_install.ps1 create mode 100644 windows/internal/vs2022_install.ps1 diff --git a/.circleci/scripts/binary_checkout.sh b/.circleci/scripts/binary_checkout.sh index f06eb8a808..b634f5c9a2 100755 --- a/.circleci/scripts/binary_checkout.sh +++ b/.circleci/scripts/binary_checkout.sh @@ -41,8 +41,8 @@ echo "export BUILDER_ROOT=${BUILDER_ROOT}" >> ${BASH_ENV} retry git clone --depth 1 https://github.com/pytorch/pytorch.git "$PYTORCH_ROOT" # Removed checking out pytorch/pytorch using CIRCLE_PR_NUMBER and CIRCLE_SHA1 as # those environment variables are tied to the host repo where the build is being -# triggered. -retry git submodule update --init --recursive --jobs 0 +# triggered. +retry git submodule update --init --recursive pushd "$PYTORCH_ROOT" echo "Using Pytorch from " git --no-pager log --max-count 1 diff --git a/.github/actions/validate-binary/action.yml b/.github/actions/validate-binary/action.yml deleted file mode 100644 index 76531010ad..0000000000 --- a/.github/actions/validate-binary/action.yml +++ /dev/null @@ -1,57 +0,0 @@ -name: 'validate-binary' -description: 'Binary Conda or Wheel Validation for Linux and MacOS' -inputs: - gpu_arch_type: - description: 'GPU arch type' - required: true - default: 'cpu' - gpu_arch_ver: - description: 'GPU arch version' - required: true - default: 'cpu' - installation: - description: 'Installation instructions' - required: true - default: '' - python_version: - description: 'Python version' - required: true - default: '3.9' - target_os: - description: 'Target OS linux or macos' - required: false - default: 'linux' -runs: - using: "composite" - steps: - - name: Checkout PyTorch builder - uses: actions/checkout@v2 - - name: Check nvidia smi - if: ${{ inputs.gpu_arch_type == 'cuda' }} - shell: bash - run: | - nvidia-smi - - name: Install Conda Linux - if: ${{ inputs.target_os == 'linux' }} - uses: conda-incubator/setup-miniconda@v2 - with: - python-version: ${{ inputs.python_version }} - auto-update-conda: true - miniconda-version: "latest" - activate-environment: testenv - - name: Install Conda MacOS - if: ${{ inputs.target_os == 'macos' }} - uses: pytorch/test-infra/.github/actions/setup-miniconda@main - - name: Install PyTorch and run tests - shell: bash - env: - GPU_ARCH_VER: ${{ inputs.gpu_arch_ver }} - GPU_ARCH_TYPE: ${{ inputs.gpu_arch_type }} - INSTALLATION: ${{ inputs.installation }} - ENV_NAME: conda-env-${{ github.run_id }} - run: | - set -ex - conda create -yp ${ENV_NAME} python=${{ inputs.python_version }} numpy - conda run -p ${ENV_NAME} $INSTALLATION - conda run -p ${ENV_NAME} python3 ./test/smoke_test/smoke_test.py - conda env remove -p ${ENV_NAME} diff --git a/.github/actions/validate-windows-binary/action.yml b/.github/actions/validate-windows-binary/action.yml deleted file mode 100644 index 7214a813bd..0000000000 --- a/.github/actions/validate-windows-binary/action.yml +++ /dev/null @@ -1,46 +0,0 @@ -name: 'validate-windows-binary' -description: 'Windows Binary Conda or Wheel Validation' -inputs: - gpu_arch_type: - description: 'GPU arch type' - required: true - default: 'cpu' - gpu_arch_ver: - description: 'GPU arch version' - required: true - default: 'cpu' - installation: - description: 'Installation instructions' - required: true - default: '' - python_version: - description: 'Python version' - required: true - default: '3.9' -runs: - using: "composite" - steps: - - name: Check nvidia smi - if: ${{ inputs.gpu_arch_type == 'cuda' }} - shell: powershell - run: | - nvidia-smi - - name: Install conda - if: ${{ inputs.gpu_arch_type == 'cpu' }} - uses: conda-incubator/setup-miniconda@v2 - with: - python-version: ${{ inputs.python_version }} - auto-update-conda: true - miniconda-version: "latest" - activate-environment: conda-env-${{ github.run_id }} - - name: Conda Install pytorch and smoke test - shell: powershell - env: - GPU_ARCH_VER: ${{ inputs.gpu_arch_ver }} - GPU_ARCH_TYPE: ${{ inputs.gpu_arch_type }} - CUDA_VER: ${{ inputs.desired_cuda }} - run: | - conda install numpy pillow python=${{ inputs.python_version }} - $install = '${{ inputs.installation }}' - Invoke-Expression $install - python ./test/smoke_test/smoke_test.py diff --git a/.github/scripts/validate_binaries.sh b/.github/scripts/validate_binaries.sh new file mode 100755 index 0000000000..6ce3dd70b3 --- /dev/null +++ b/.github/scripts/validate_binaries.sh @@ -0,0 +1,50 @@ +if [[ ${MATRIX_PACKAGE_TYPE} == "libtorch" ]]; then + curl ${MATRIX_INSTALLATION} -o libtorch.zip + unzip libtorch.zip +else + #special case for Python 3.11 + if [[ ${MATRIX_PYTHON_VERSION} == '3.11' ]]; then + conda create -y -n ${ENV_NAME} python=${MATRIX_PYTHON_VERSION} + conda activate ${ENV_NAME} + + INSTALLATION=${MATRIX_INSTALLATION/"-c pytorch"/"-c malfet -c pytorch"} + INSTALLATION=${INSTALLATION/"pytorch-cuda"/"pytorch-${MATRIX_CHANNEL}::pytorch-cuda"} + INSTALLATION=${INSTALLATION/"conda install"/"conda install -y"} + + eval $INSTALLATION + python ./test/smoke_test/smoke_test.py + conda deactivate + conda env remove -n ${ENV_NAME} + else + + + + # Special case Pypi installation package, only applicable to linux nightly CUDA 11.7 builds, wheel package + if [[ ${TARGET_OS} == 'linux' && ${MATRIX_GPU_ARCH_VERSION} == '11.7' && ${MATRIX_PACKAGE_TYPE} == 'manywheel' && ${MATRIX_CHANNEL} != 'nightly' ]]; then + conda create -yp ${ENV_NAME}_pypi python=${MATRIX_PYTHON_VERSION} numpy ffmpeg + INSTALLATION_PYPI=${MATRIX_INSTALLATION/"cu117"/"cu117_pypi_cudnn"} + INSTALLATION_PYPI=${INSTALLATION_PYPI/"torchvision torchaudio"/""} + INSTALLATION_PYPI=${INSTALLATION_PYPI/"index-url"/"extra-index-url"} + conda run -p ${ENV_NAME}_pypi ${INSTALLATION_PYPI} + conda run -p ${ENV_NAME}_pypi python ./test/smoke_test/smoke_test.py --package torchonly + conda deactivate + conda env remove -p ${ENV_NAME}_pypi + fi + + # Please note ffmpeg is required for torchaudio, see https://github.com/pytorch/pytorch/issues/96159 + conda create -y -n ${ENV_NAME} python=${MATRIX_PYTHON_VERSION} numpy ffmpeg + conda activate ${ENV_NAME} + INSTALLATION=${MATRIX_INSTALLATION/"conda install"/"conda install -y"} + eval $INSTALLATION + + if [[ ${TARGET_OS} == 'linux' ]]; then + export CONDA_LIBRARY_PATH="$(dirname $(which python))/../lib" + export LD_LIBRARY_PATH=$CONDA_LIBRARY_PATH:$LD_LIBRARY_PATH + ${PWD}/check_binary.sh + fi + + python ./test/smoke_test/smoke_test.py + conda deactivate + conda env remove -n ${ENV_NAME} + fi +fi diff --git a/.github/workflows/build-conda-images.yml b/.github/workflows/build-conda-images.yml index 92567d7bd3..43626533e6 100644 --- a/.github/workflows/build-conda-images.yml +++ b/.github/workflows/build-conda-images.yml @@ -19,19 +19,19 @@ env: DOCKER_BUILDKIT: 1 DOCKER_ID: ${{ secrets.DOCKER_ID }} DOCKER_TOKEN: ${{ secrets.DOCKER_TOKEN }} - WITH_PUSH: ${{ github.event_name == 'push' }} + WITH_PUSH: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} jobs: build-docker: - runs-on: linux.2xlarge + runs-on: ubuntu-22.04 strategy: matrix: - cuda_version: ["10.2", "11.3", "11.5", "11.6", "11.7", "cpu"] + cuda_version: ["11.6", "11.7", "11.8", "cpu"] env: CUDA_VERSION: ${{ matrix.cuda_version }} steps: - name: Checkout PyTorch builder - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Authenticate if WITH_PUSH run: | if [[ "${WITH_PUSH}" == true ]]; then diff --git a/.github/workflows/build-libtorch-images.yml b/.github/workflows/build-libtorch-images.yml index 9526434e4e..49069557a9 100644 --- a/.github/workflows/build-libtorch-images.yml +++ b/.github/workflows/build-libtorch-images.yml @@ -21,20 +21,20 @@ env: DOCKER_BUILDKIT: 1 DOCKER_ID: ${{ secrets.DOCKER_ID }} DOCKER_TOKEN: ${{ secrets.DOCKER_TOKEN }} - WITH_PUSH: ${{ github.event_name == 'push' }} + WITH_PUSH: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} jobs: build-docker-cuda: - runs-on: ubuntu-18.04 + runs-on: ubuntu-22.04 strategy: matrix: - cuda_version: ["11.7", "11.6", "11.5", "11.3", "10.2"] + cuda_version: ["11.8", "11.7", "11.6"] env: GPU_ARCH_TYPE: cuda GPU_ARCH_VERSION: ${{ matrix.cuda_version }} steps: - name: Checkout PyTorch builder - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Authenticate if WITH_PUSH run: | if [[ "${WITH_PUSH}" == true ]]; then @@ -44,16 +44,16 @@ jobs: run: | libtorch/build_docker.sh build-docker-rocm: - runs-on: ubuntu-18.04 + runs-on: ubuntu-22.04 strategy: matrix: - rocm_version: ["5.1.1", "5.2"] + rocm_version: ["5.3", "5.4.2"] env: GPU_ARCH_TYPE: rocm GPU_ARCH_VERSION: ${{ matrix.rocm_version }} steps: - name: Checkout PyTorch - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Authenticate if WITH_PUSH run: | if [[ "${WITH_PUSH}" == true ]]; then @@ -63,10 +63,10 @@ jobs: run: | libtorch/build_docker.sh build-docker-cpu: - runs-on: ubuntu-18.04 + runs-on: ubuntu-22.04 steps: - name: Checkout PyTorch - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Authenticate if WITH_PUSH run: | if [[ "${WITH_PUSH}" == true ]]; then diff --git a/.github/workflows/build-llvm-images.yml b/.github/workflows/build-llvm-images.yml index 5b24abf5fa..a89230891b 100644 --- a/.github/workflows/build-llvm-images.yml +++ b/.github/workflows/build-llvm-images.yml @@ -17,7 +17,7 @@ env: DOCKER_BUILDKIT: 1 DOCKER_ID: ${{ secrets.DOCKER_ID }} DOCKER_TOKEN: ${{ secrets.DOCKER_TOKEN }} - WITH_PUSH: ${{ github.event_name == 'push' }} + WITH_PUSH: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} FORCE_PUSH: yes jobs: @@ -25,7 +25,7 @@ jobs: runs-on: linux.2xlarge steps: - name: Checkout PyTorch builder - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Authenticate if WITH_PUSH run: | if [[ "${WITH_PUSH}" == true ]]; then diff --git a/.github/workflows/build-magma-linux.yml b/.github/workflows/build-magma-linux.yml index 655d02c6ee..eb1d67a70a 100644 --- a/.github/workflows/build-magma-linux.yml +++ b/.github/workflows/build-magma-linux.yml @@ -30,10 +30,10 @@ jobs: runs-on: linux.2xlarge strategy: matrix: - cuda_version: ["117", "116", "115"] + cuda_version: ["118", "117", "116"] steps: - name: Checkout PyTorch builder - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Build Magma Cuda working-directory: magma run: | @@ -54,7 +54,7 @@ jobs: run: | conda install -y conda-build anaconda-client - name: Push MAGMA to anaconda - if: ${{ github.event_name == 'push' }} + if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} run: | anaconda --token $ANACONDA_TOKEN upload -u pytorch --force magma/output/linux-64/magma-cuda*.bz2 env: diff --git a/.github/workflows/build-magma-windows.yml b/.github/workflows/build-magma-windows.yml index 87fdb22c5a..5ad6ba29a6 100644 --- a/.github/workflows/build-magma-windows.yml +++ b/.github/workflows/build-magma-windows.yml @@ -17,14 +17,14 @@ jobs: runs-on: windows-2019 strategy: matrix: - cuda_version: ["117", "116"] + cuda_version: ["118", "117", "116"] config: ["Release", "Debug"] env: CUDA_VERSION: ${{ matrix.cuda_version }} CONFIG: ${{ matrix.config }} steps: - name: Checkout pytorch/builder - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Enable MSVC dev commands to enable cl.exe # FYI incompatible with shell: bash uses: ilammy/msvc-dev-cmd@dd5e2fa0a7de1e7929605d9ecc020e749d9856a3 - name: Install CUDA Toolkit @@ -36,9 +36,9 @@ jobs: with: path: magma_*_cuda*_*.7z push-windows-magma: - if: ${{ github.event_name == 'push' }} + if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} environment: magma - runs-on: ubuntu-18.04 + runs-on: ubuntu-22.04 needs: build-windows-magma steps: - name: Download all artifacts diff --git a/.github/workflows/build-manywheel-images.yml b/.github/workflows/build-manywheel-images.yml index b62507bbd0..153f501bc7 100644 --- a/.github/workflows/build-manywheel-images.yml +++ b/.github/workflows/build-manywheel-images.yml @@ -7,12 +7,14 @@ on: paths: - .github/workflows/build-manywheel-images.yml - manywheel/Dockerfile + - manywheel/Dockerfile_cxx11-abi - manywheel/build_docker.sh - 'common/*' pull_request: paths: - .github/workflows/build-manywheel-images.yml - manywheel/Dockerfile + - manywheel/Dockerfile_cxx11-abi - 'common/*' - manywheel/build_docker.sh @@ -21,20 +23,20 @@ env: DOCKER_BUILDKIT: 1 DOCKER_ID: ${{ secrets.DOCKER_ID }} DOCKER_TOKEN: ${{ secrets.DOCKER_TOKEN }} - WITH_PUSH: ${{ github.event_name == 'push' }} + WITH_PUSH: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }} jobs: build-docker-cuda: - runs-on: ubuntu-18.04 + runs-on: ubuntu-22.04 strategy: matrix: - cuda_version: ["11.7", "11.6", "11.5", "11.3", "10.2"] + cuda_version: ["11.8", "11.7", "11.6"] env: GPU_ARCH_TYPE: cuda GPU_ARCH_VERSION: ${{ matrix.cuda_version }} steps: - name: Checkout PyTorch builder - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Authenticate if WITH_PUSH run: | if [[ "${WITH_PUSH}" == true ]]; then @@ -44,16 +46,16 @@ jobs: run: | manywheel/build_docker.sh build-docker-rocm: - runs-on: ubuntu-18.04 + runs-on: ubuntu-22.04 strategy: matrix: - rocm_version: ["5.1.1", "5.2"] + rocm_version: ["5.3", "5.4.2"] env: GPU_ARCH_TYPE: rocm GPU_ARCH_VERSION: ${{ matrix.rocm_version }} steps: - name: Checkout PyTorch - uses: actions/checkout@v2 + uses: actions/checkout@v3 - name: Authenticate if WITH_PUSH run: | if [[ "${WITH_PUSH}" == true ]]; then @@ -63,10 +65,25 @@ jobs: run: | manywheel/build_docker.sh build-docker-cpu: - runs-on: ubuntu-18.04 + runs-on: ubuntu-22.04 steps: - name: Checkout PyTorch - uses: actions/checkout@v2 + uses: actions/checkout@v3 + - name: Authenticate if WITH_PUSH + run: | + if [[ "${WITH_PUSH}" == true ]]; then + echo "${DOCKER_TOKEN}" | docker login -u "${DOCKER_ID}" --password-stdin + fi + - name: Build Docker Image + run: | + manywheel/build_docker.sh + build-docker-cpu-cxx11-abi: + runs-on: ubuntu-22.04 + env: + GPU_ARCH_TYPE: cpu-cxx11-abi + steps: + - name: Checkout PyTorch + uses: actions/checkout@v3 - name: Authenticate if WITH_PUSH run: | if [[ "${WITH_PUSH}" == true ]]; then diff --git a/.github/workflows/test-validate-domain-library.yml b/.github/workflows/test-validate-domain-library.yml new file mode 100644 index 0000000000..6c651e709a --- /dev/null +++ b/.github/workflows/test-validate-domain-library.yml @@ -0,0 +1,19 @@ +name: Test validate domain library + +on: + pull_request: + paths: + - .github/workflows/validate-domain-library.yml + - .github/workflows/test-validate-domain-library.yml + workflow_dispatch: + +jobs: + test-validate-domain-library: + uses: ./.github/workflows/validate-domain-library.yml + with: + package_type: "conda,wheel" + os: "all" + channel: "release" + repository: "pytorch/builder" + ref: main + smoke_test: "echo test" diff --git a/.github/workflows/validate-binaries.yml b/.github/workflows/validate-binaries.yml new file mode 100644 index 0000000000..4ae2605386 --- /dev/null +++ b/.github/workflows/validate-binaries.yml @@ -0,0 +1,90 @@ +name: Validate binaries + +# A reusable workflow that triggers a set of jobs that perform a smoke test / validation of pytorch binaries. +# Optionally restricts validation to the specified OS and channel. +# For the details about parameter values, see: +# pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main +# For an example of the `workflow_call` usage see: +# https://github.com/pytorch/builder/pull/1144 +on: + workflow_call: + inputs: + os: + description: "Operating system to generate for (linux, windows, macos, macos-arm64)" + required: true + type: string + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: string + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + limit-win-builds: + description: "Limit windows builds to single python/cuda config" + default: "disable" + type: string + workflow_dispatch: + inputs: + os: + description: "Operating system to generate for (linux, windows, macos, macos-arm64)" + required: true + type: choice + default: all + options: + - windows + - linux + - macos + - all + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: choice + default: all + options: + - release + - nightly + - test + - all + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + limit-win-builds: + description: "Limit windows builds to single python/cuda config" + default: "disable" + required: false + type: string + +jobs: + win: + if: inputs.os == 'windows' || inputs.os == 'all' + uses: ./.github/workflows/validate-windows-binaries.yml + with: + channel: ${{ inputs.channel }} + ref: ${{ inputs.ref || github.ref }} + limit-win-builds: ${{ inputs.limit-win-builds }} + + linux: + if: inputs.os == 'linux' || inputs.os == 'all' + uses: ./.github/workflows/validate-linux-binaries.yml + with: + channel: ${{ inputs.channel }} + ref: ${{ inputs.ref || github.ref }} + + mac: + if: inputs.os == 'macos' || inputs.os == 'all' + uses: ./.github/workflows/validate-macos-binaries.yml + with: + channel: ${{ inputs.channel }} + ref: ${{ inputs.ref || github.ref }} + + mac-arm64: + if: inputs.os == 'macos' || inputs.os == 'all' + uses: ./.github/workflows/validate-macos-arm64-binaries.yml + with: + channel: ${{ inputs.channel }} + ref: ${{ inputs.ref || github.ref }} diff --git a/.github/workflows/validate-domain-library.yml b/.github/workflows/validate-domain-library.yml new file mode 100644 index 0000000000..149b8335b4 --- /dev/null +++ b/.github/workflows/validate-domain-library.yml @@ -0,0 +1,153 @@ +name: Validate domain libary + +# A reusable workflow that triggers a set of jobs that perform a smoke test / validation of pytorch binaries. +# Optionally restricts validation to the specified OS and channel. +# For the details about parameter values, see: +# pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main +on: + workflow_call: + inputs: + os: + description: "Operating system to generate for (linux, windows, macos, macos-arm64)" + required: false + type: string + default: "all" + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: string + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + package_type: + description: "Package type (conda, wheel, all)" + required: false + type: string + default: "all" + repository: + description: "Path to repository to checkout" + required: true + type: string + smoke_test: + description: "Path to a smoke test script" + required: true + type: string + with_cuda: + description: "With cuda enable/disable" + required: false + type: string + default: disable + +jobs: + generate-linux-matrix: + if: (inputs.os == 'linux' || inputs.os == 'all') + uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main + with: + package-type: ${{ inputs.package_type }} + os: linux + channel: ${{ inputs.channel }} + with-cuda: ${{ inputs.with_cuda }} + generate-windows-matrix: + if: (inputs.os == 'windows' || inputs.os == 'all') + uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main + with: + package-type: ${{ inputs.package_type }} + os: windows + channel: ${{ inputs.channel }} + with-cuda: ${{ inputs.with_cuda }} + generate-macos-matrix: + if: (inputs.os == 'macos' || inputs.os == 'all') + uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main + with: + package-type: ${{ inputs.package_type }} + os: macos + channel: ${{ inputs.channel }} + with-cuda: ${{ inputs.with_cuda }} + generate-macos-arm64-matrix: + if: (inputs.os == 'macos-arm64' || inputs.os == 'all') + uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main + with: + package-type: ${{ inputs.package_type }} + os: macos-arm64 + channel: ${{ inputs.channel }} + with-cuda: ${{ inputs.with_cuda }} + validate-linux: + if: (inputs.os == 'linux' || inputs.os == 'all') + needs: generate-linux-matrix + strategy: + matrix: ${{ fromJson(needs.generate-linux-matrix.outputs.matrix) }} + fail-fast: false + uses: pytorch/test-infra/.github/workflows/linux_job.yml@main + name: "linux-${{ matrix.package_type }}-${{ matrix.python_version }}-${{ matrix.desired_cuda }}" + with: + runner: ${{ matrix.validation_runner }} + repository: ${{ inputs.repository }} + ref: ${{ inputs.ref || github.ref }} + job-name: "linux-${{ matrix.package_type }}-${{ matrix.python_version }}-${{ matrix.desired_cuda }}" + binary-matrix: ${{ toJSON(matrix) }} + script: | + set -ex + export ENV_NAME="conda-env-${{ github.run_id }}" + export SMOKE_TEST="${{ inputs.smoke_test }}" + eval $SMOKE_TEST + validate-windows: + if: (inputs.os == 'windows' || inputs.os == 'all') + needs: generate-windows-matrix + strategy: + matrix: ${{ fromJson(needs.generate-windows-matrix.outputs.matrix) }} + fail-fast: false + uses: pytorch/test-infra/.github/workflows/windows_job.yml@main + name: "windows-${{ matrix.package_type }}-${{ matrix.python_version }}-${{ matrix.desired_cuda }}" + with: + runner: ${{ matrix.validation_runner }} + repository: ${{ inputs.repository }} + ref: ${{ inputs.ref || github.ref }} + job-name: "windows-${{ matrix.package_type }}-${{ matrix.python_version }}-${{ matrix.desired_cuda }}" + binary-matrix: ${{ toJSON(matrix) }} + script: | + set -ex + export ENV_NAME="conda-env-${{ github.run_id }}" + export SMOKE_TEST="${{ inputs.smoke_test }}" + export TARGET_OS="windows" + eval $SMOKE_TEST + validate-macos: + if: (inputs.os == 'macos' || inputs.os == 'all') + needs: generate-macos-matrix + strategy: + matrix: ${{ fromJson(needs.generate-macos-matrix.outputs.matrix) }} + fail-fast: false + uses: pytorch/test-infra/.github/workflows/macos_job.yml@main + name: "macos-${{ matrix.package_type }}-${{ matrix.python_version }}-${{ matrix.desired_cuda }}" + with: + runner: ${{ matrix.validation_runner }} + repository: ${{ inputs.repository }} + ref: ${{ inputs.ref || github.ref }} + job-name: "macos-${{ matrix.package_type }}-${{ matrix.python_version }}-${{ matrix.desired_cuda }}" + binary-matrix: ${{ toJSON(matrix) }} + script: | + set -ex + export ENV_NAME="conda-env-${{ github.run_id }}" + export TARGET_OS="macos" + export SMOKE_TEST="${{ inputs.smoke_test }}" + eval $SMOKE_TEST + validate-macos-arm64: + if: (inputs.os == 'macos-arm64' || inputs.os == 'all') + needs: generate-macos-matrix + strategy: + matrix: ${{ fromJson(needs.generate-macos-arm64-matrix.outputs.matrix) }} + fail-fast: false + uses: pytorch/test-infra/.github/workflows/macos_job.yml@main + name: "macos-${{ matrix.package_type }}-${{ matrix.python_version }}-${{ matrix.desired_cuda }}" + with: + runner: ${{ matrix.validation_runner }} + repository: ${{ inputs.repository }} + ref: ${{ inputs.ref || github.ref }} + job-name: "macos-arm64-${{ matrix.package_type }}-${{ matrix.python_version }}-${{ matrix.desired_cuda }}" + binary-matrix: ${{ toJSON(matrix) }} + script: | + set -ex + export ENV_NAME="conda-env-${{ github.run_id }}" + export TARGET_OS="macos-arm64" + eval $SMOKE_TEST diff --git a/.github/workflows/validate-linux-binaries.yml b/.github/workflows/validate-linux-binaries.yml index 43c1a484d5..438062f91a 100644 --- a/.github/workflows/validate-linux-binaries.yml +++ b/.github/workflows/validate-linux-binaries.yml @@ -1,79 +1,58 @@ name: Validate linux binaries on: - push: - branches: - main - paths: - - .github/workflows/validate-linux-binaries.yml - pull_request: - paths: - - .github/workflows/validate-linux-binaries.yml + workflow_call: + inputs: + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: string + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + workflow_dispatch: + inputs: + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: choice + options: + - release + - nightly + - test + - all + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + jobs: - generate-conda-matrix: - uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main - with: - package-type: conda - os: linux - channel: nightly - generate-wheel-matrix: - uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main - with: - package-type: wheel - os: linux - channel: nightly - generate-libtorch-matrix: + generate-linux-matrix: uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main with: - package-type: libtorch + package-type: all os: linux - channel: nightly - validate-linux-binaries-conda: - needs: generate-conda-matrix - strategy: - matrix: - ${{ fromJson(needs.generate-conda-matrix.outputs.matrix) }} - fail-fast: false - runs-on: ${{ matrix.validation_runner }} - steps: - - name: Validate binary conda - uses: pytorch/builder/.github/actions/validate-binary@main - with: - gpu_arch_type: ${{ matrix.gpu_arch_type }} - gpu_arch_ver: ${{ matrix.gpu_arch_version }} - installation: ${{ matrix.installation }} - python_version: ${{ matrix.python_version }} - validate-linux-binaries-wheels: - needs: generate-wheel-matrix - strategy: - matrix: - ${{ fromJson(needs.generate-wheel-matrix.outputs.matrix) }} - fail-fast: false - runs-on: ${{ matrix.validation_runner }} - steps: - - name: Validate binary wheel - uses: pytorch/builder/.github/actions/validate-binary@main - with: - gpu_arch_type: ${{ matrix.gpu_arch_type }} - gpu_arch_ver: ${{ matrix.gpu_arch_version }} - installation: ${{ matrix.installation }} - python_version: ${{ matrix.python_version }} - validate-linux-libtorch-binaries: - needs: generate-libtorch-matrix + channel: ${{ inputs.channel }} + + linux: + needs: generate-linux-matrix strategy: - matrix: - ${{ fromJson(needs.generate-libtorch-matrix.outputs.matrix) }} + matrix: ${{ fromJson(needs.generate-linux-matrix.outputs.matrix) }} fail-fast: false - runs-on: "ubuntu-20.04" - env: - PYTHON_VERSION: ${{ matrix.python_version }} - steps: - - name: Install pytorch and smoke test - env: - INSTALLATION: ${{ matrix.installation }} - ENV_NAME: conda-env-${{ github.run_id }} - run: | - sudo apt-get install unzip -y - set -ex - curl ${INSTALLATION} -o libtorch.zip - unzip libtorch.zip + uses: pytorch/test-infra/.github/workflows/linux_job.yml@main + name: ${{ matrix.build_name }} + with: + runner: ${{ matrix.validation_runner }} + repository: "pytorch/builder" + ref: ${{ inputs.ref || github.ref }} + job-name: ${{ matrix.build_name }} + binary-matrix: ${{ toJSON(matrix) }} + script: | + set -ex + export ENV_NAME="conda-env-${{ github.run_id }}" + export TARGET_OS="linux" + eval "$(conda shell.bash hook)" + source ./.github/scripts/validate_binaries.sh diff --git a/.github/workflows/validate-macos-arm64-binaries.yml b/.github/workflows/validate-macos-arm64-binaries.yml new file mode 100644 index 0000000000..f321022d42 --- /dev/null +++ b/.github/workflows/validate-macos-arm64-binaries.yml @@ -0,0 +1,56 @@ +name: Validate MacOS ARM64 Binaries + +on: + workflow_call: + inputs: + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: string + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + workflow_dispatch: + inputs: + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: choice + options: + - release + - nightly + - test + - all + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + +jobs: + generate-macos-arm64-matrix: + uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main + with: + package-type: all + os: macos-arm64 + channel: ${{ inputs.channel }} + macos-arm64: + needs: generate-macos-arm64-matrix + strategy: + matrix: ${{ fromJson(needs.generate-macos-arm64-matrix.outputs.matrix) }} + fail-fast: false + uses: pytorch/test-infra/.github/workflows/macos_job.yml@main + name: ${{ matrix.build_name }} + with: + runner: ${{ matrix.validation_runner }} + repository: "pytorch/builder" + ref: ${{ inputs.ref || github.ref }} + job-name: ${{ matrix.build_name }} + binary-matrix: ${{ toJSON(matrix) }} + script: | + set -ex + export ENV_NAME="conda-env-${{ github.run_id }}" + export TARGET_OS="macos-arm64" + source ./.github/scripts/validate_binaries.sh diff --git a/.github/workflows/validate-macos-binaries.yml b/.github/workflows/validate-macos-binaries.yml index 3bc3ea0cdc..0e3f38ff86 100644 --- a/.github/workflows/validate-macos-binaries.yml +++ b/.github/workflows/validate-macos-binaries.yml @@ -1,96 +1,56 @@ name: Validate MacOS Binaries on: - pull_request: - paths: - - .github/workflows/validate-macos-binaries.yml + workflow_call: + inputs: + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: string + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + workflow_dispatch: + inputs: + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: choice + options: + - release + - nightly + - test + - all + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + jobs: - generate-arm64-conda-matrix: - uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main - with: - package-type: conda - os: macos-arm64 - channel: all - generate-arm64-wheel-matrix: - uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main - with: - package-type: wheel - os: macos-arm64 - channel: all - generate-x86_64-conda-matrix: - uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main - with: - package-type: conda - os: macos-x86_64 - channel: all - generate-x86_64-wheel-matrix: + generate-macos-matrix: uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main with: - package-type: wheel - os: macos-x86_64 - channel: all - - validate-macos-arm64-binaries-conda: - needs: generate-arm64-conda-matrix + package-type: all + os: macos + channel: ${{ inputs.channel }} + macos: + needs: generate-macos-matrix strategy: - matrix: - ${{ fromJson(needs.generate-arm64-conda-matrix.outputs.matrix) }} + matrix: ${{ fromJson(needs.generate-macos-matrix.outputs.matrix) }} fail-fast: false - runs-on: ${{ matrix.validation_runner }} - steps: - - name: Validate binary conda - uses: pytorch/builder/.github/actions/validate-binary@main - with: - gpu_arch_type: ${{ matrix.gpu_arch_type }} - gpu_arch_ver: ${{ matrix.gpu_arch_version }} - installation: ${{ matrix.installation }} - python_version: ${{ matrix.python_version }} - target_os: macos - validate-macos-arm64-binaries-wheel: - needs: generate-arm64-wheel-matrix - strategy: - matrix: - ${{ fromJson(needs.generate-arm64-wheel-matrix.outputs.matrix) }} - fail-fast: false - runs-on: ${{ matrix.validation_runner }} - steps: - - name: Validate binary wheel - uses: pytorch/builder/.github/actions/validate-binary@main - with: - gpu_arch_type: ${{ matrix.gpu_arch_type }} - gpu_arch_ver: ${{ matrix.gpu_arch_version }} - installation: ${{ matrix.installation }} - python_version: ${{ matrix.python_version }} - target_os: macos - validate-macos-x86_64-binaries-conda: - needs: generate-x86_64-conda-matrix - strategy: - matrix: - ${{ fromJson(needs.generate-x86_64-conda-matrix.outputs.matrix) }} - fail-fast: false - runs-on: ${{ matrix.validation_runner }} - steps: - - name: Validate binary conda - uses: pytorch/builder/.github/actions/validate-binary@main - with: - gpu_arch_type: ${{ matrix.gpu_arch_type }} - gpu_arch_ver: ${{ matrix.gpu_arch_version }} - installation: ${{ matrix.installation }} - python_version: ${{ matrix.python_version }} - target_os: macos - validate-macos-x86_64-binaries-wheel: - needs: generate-x86_64-wheel-matrix - strategy: - matrix: - ${{ fromJson(needs.generate-x86_64-wheel-matrix.outputs.matrix) }} - fail-fast: false - runs-on: ${{ matrix.validation_runner }} - steps: - - name: Validate binary wheel - uses: pytorch/builder/.github/actions/validate-binary@main - with: - gpu_arch_type: ${{ matrix.gpu_arch_type }} - gpu_arch_ver: ${{ matrix.gpu_arch_version }} - installation: ${{ matrix.installation }} - python_version: ${{ matrix.python_version }} - target_os: macos + uses: pytorch/test-infra/.github/workflows/macos_job.yml@main + name: ${{ matrix.build_name }} + with: + runner: ${{ matrix.validation_runner }} + repository: "pytorch/builder" + ref: ${{ inputs.ref || github.ref }} + job-name: ${{ matrix.build_name }} + binary-matrix: ${{ toJSON(matrix) }} + script: | + set -ex + export ENV_NAME="conda-env-${{ github.run_id }}" + export TARGET_OS="macos" + source ./.github/scripts/validate_binaries.sh diff --git a/.github/workflows/validate-nightly-binaries.yml b/.github/workflows/validate-nightly-binaries.yml new file mode 100644 index 0000000000..c252e0433b --- /dev/null +++ b/.github/workflows/validate-nightly-binaries.yml @@ -0,0 +1,35 @@ +# Scheduled validation of the nightly binaries +name: cron + +on: + schedule: + # At 2:30 pm UTC (7:30 am PDT) + - cron: "30 14 * * *" + # Have the ability to trigger this job manually through the API + workflow_dispatch: + push: + branches: + - main + paths: + - .github/workflows/validate-nightly-binaries.yml + - .github/workflows/validate-linux-binaries.yml + - .github/workflows/validate-windows-binaries.yml + - .github/workflows/validate-macos-binaries.yml + - .github/workflows/validate-macos-arm64-binaries.yml + - test/smoke_test/* + pull_request: + paths: + - .github/workflows/validate-nightly-binaries.yml + - .github/workflows/validate-linux-binaries.yml + - .github/workflows/validate-windows-binaries.yml + - .github/workflows/validate-macos-binaries.yml + - .github/workflows/validate-macos-arm64-binaries.yml + - .github/scripts/validate_binaries.sh + - test/smoke_test/* +jobs: + nightly: + uses: ./.github/workflows/validate-binaries.yml + with: + channel: nightly + os: all + limit-win-builds: enable diff --git a/.github/workflows/validate-nightly-pypi-wheel-binary-size.yml b/.github/workflows/validate-nightly-pypi-wheel-binary-size.yml new file mode 100644 index 0000000000..a995ec817a --- /dev/null +++ b/.github/workflows/validate-nightly-pypi-wheel-binary-size.yml @@ -0,0 +1,26 @@ +name: Validate Nightly PyPI Wheel Binary Size +on: + pull_request: + paths: + - .github/workflows/validate-nightly-pypi-wheel-binary-size.yml + workflow_dispatch: + schedule: + # At 2:30 pm UTC (7:30 am PDT) + - cron: "30 14 * * *" + +jobs: + nightly-pypi-binary-size-validation: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + repository: pytorch/test-infra + - name: Install requirements + run: | + pip3 install -r tools/binary_size_validation/requirements.txt + - name: Run validation + run: | + python tools/binary_size_validation/binary_size_validation.py \ + --url https://download.pytorch.org/whl/nightly/torch/ \ + --include "pypi" --only-latest-version --threshold 750 \ No newline at end of file diff --git a/.github/workflows/validate-release-binaries.yml b/.github/workflows/validate-release-binaries.yml new file mode 100644 index 0000000000..9549e1e33e --- /dev/null +++ b/.github/workflows/validate-release-binaries.yml @@ -0,0 +1,27 @@ +# Scheduled validation of the release binaries +name: cron + +on: + schedule: + # At 3 am and 2 pm UTC (7 am and 8 pm PDT) + - cron: "0 3,14 * * *" + # Have the ability to trigger this job manually through the API + workflow_dispatch: + push: + branches: + - main + paths: + - .github/workflows/validate-release-binaries.yml + - .github/workflows/validate-linux-binaries.yml + - .github/workflows/validate-windows-binaries.yml + - .github/workflows/validate-macos-binaries.yml + - .github/workflows/validate-macos-arm64-binaries.yml + - test/smoke_test/* + +jobs: + release: + uses: ./.github/workflows/validate-binaries.yml + with: + channel: release + os: all + limit-win-builds: enable diff --git a/.github/workflows/validate-repackaged-binary-sizes.yml b/.github/workflows/validate-repackaged-binary-sizes.yml new file mode 100644 index 0000000000..695c68d3aa --- /dev/null +++ b/.github/workflows/validate-repackaged-binary-sizes.yml @@ -0,0 +1,88 @@ +name: Validate manywheel binaries + +# This workflow validates the size of the manywheel binaries after repackaging for PyPi +# Specify the direct URLs to the binaries (from https://download.pytorch.org/whl/test/torch/) in the matrix +# along with the python version. +# +# The workflow will: +# * download the binaries, +# * run release/pypi/prep_binary_for_pypi.sh +# * run smoke tests on the repackaged binaries +# * display the size before and after repackaging as the workflow annotation +# * optionally upload the repackaged binaries as artifacts (for debug or promotion) + +on: + pull_request: + paths: + - .github/workflows/validate-repackaged-binary-sizes.yml + - release/pypi/prep_binary_for_pypi.sh + +jobs: + validate-binary-size: + strategy: + fail-fast: false + matrix: + whl: + - url: https://download.pytorch.org/whl/test/cu117_pypi_cudnn/torch-1.13.1%2Bcu117.with.pypi.cudnn-cp310-cp310-linux_x86_64.whl + python: "3.10" # python version to use for smoke tests + upload_artifact: false # upload the repackaged binary as an artifact + - url: https://download.pytorch.org/whl/test/cu117_pypi_cudnn/torch-1.13.1%2Bcu117.with.pypi.cudnn-cp37-cp37m-linux_x86_64.whl + python: "3.7" + artifact: false + - url: https://download.pytorch.org/whl/test/cu117_pypi_cudnn/torch-1.13.1%2Bcu117.with.pypi.cudnn-cp38-cp38-linux_x86_64.whl + python: "3.8" + artifact: false + - url: https://download.pytorch.org/whl/test/cu117_pypi_cudnn/torch-1.13.1%2Bcu117.with.pypi.cudnn-cp39-cp39-linux_x86_64.whl + python: "3.9" + artifact: false + # - url: https://download.pytorch.org/whl/test/cu117_pypi_cudnn/torch-1.13.1%2Bcu117.with.pypi.cudnn-cp311-cp311-linux_x86_64.whl + # python: "3.11" + # artifact: false + + uses: pytorch/test-infra/.github/workflows/linux_job.yml@main + with: + runner: linux.4xlarge.nvidia.gpu + job-name: "Validate binary size" + upload-artifact: ${{ matrix.whl.upload_artifact == 'true' && 'repackaged-binary' || '' }} + script: | + set -ex + export ENV_NAME="conda-env-${{ github.run_id }}" + export GPU_ARCH_VER="11.7" + export GPU_ARCH_TYPE="cuda" + export CUDA_VER="11.7" + export DESIRED_PYTHON="${{ matrix.whl.python }}" + export DESIRED_CUDA="cu117" + export PACKAGE_TYPE="wheel" + export TARGET_OS="linux" + export INSTALLATION="" + + # install zip + sudo yum install zip -y + + # install patchelf + chmod a+x common/install_patchelf.sh + sudo common/install_patchelf.sh + + # download torch whl + wget ${{ matrix.whl.url }} + FILENAME=$(ls -1 *.whl | head -n 1) + SIZE_BEFORE=$(du -h $FILENAME | cut -f1) + + # repackage into manywheel + release/pypi/prep_binary_for_pypi.sh $FILENAME + + NEW_FILENAME=$(ls -1 *.whl | head -n 1) + echo "::notice:: $FILENAME before: $SIZE_BEFORE after: $(du -h $NEW_FILENAME | cut -f1)" + + # cp to ${RUNNER_ARTIFACT_DIR} + cp $NEW_FILENAME ${RUNNER_ARTIFACT_DIR}/ + + # create conda env + conda create -y -n $ENV_NAME python=$DESIRED_PYTHON + conda activate $ENV_NAME + + # install torch + pip install numpy pillow $NEW_FILENAME + + # run smoke test + python ./test/smoke_test/smoke_test.py --package=torchonly \ No newline at end of file diff --git a/.github/workflows/validate-windows-binaries.yml b/.github/workflows/validate-windows-binaries.yml index 1dad91db06..6833e55b20 100644 --- a/.github/workflows/validate-windows-binaries.yml +++ b/.github/workflows/validate-windows-binaries.yml @@ -1,80 +1,69 @@ -name: Validate binary images +name: Validate Windows binary images on: - push: - branches: - main - paths: - - .github/workflows/validate-windows-binaries.yml - pull_request: - paths: - - .github/workflows/validate-windows-binaries.yml + workflow_call: + inputs: + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: string + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + limit-win-builds: + description: "Limit windows builds to single python/cuda config" + default: "disable" + type: string + workflow_dispatch: + inputs: + channel: + description: "Channel to use (nightly, test, release, all)" + required: true + type: choice + options: + - release + - nightly + - test + - all + ref: + description: 'Reference to checkout, defaults to empty' + default: "" + required: false + type: string + limit-win-builds: + description: "Limit windows builds to single python/cuda config" + default: "disable" + required: false + type: string + jobs: - generate-conda-matrix: - uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main - with: - package-type: conda - os: windows - channel: nightly - generate-wheel-matrix: - uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main - with: - package-type: wheel - os: windows - channel: nightly - generate-libtorch-matrix: + generate-windows-matrix: uses: pytorch/test-infra/.github/workflows/generate_binary_build_matrix.yml@main with: - package-type: libtorch + package-type: all os: windows - channel: nightly - validate-windows-binaries-conda: - needs: generate-conda-matrix - strategy: - matrix: - ${{ fromJson(needs.generate-conda-matrix.outputs.matrix) }} - fail-fast: false - runs-on: ${{ matrix.validation_runner }} - steps: - - name: Checkout PyTorch builder - uses: actions/checkout@v2 - - name: Validate binary conda - uses: ./.github/actions/validate-windows-binary - with: - gpu_arch_type: ${{ matrix.gpu_arch_type }} - gpu_arch_ver: ${{ matrix.gpu_arch_version }} - installation: ${{ matrix.installation }} - python_version: ${{ matrix.python_version }} - validate-windows-binaries-wheel: - needs: generate-wheel-matrix - strategy: - matrix: - ${{ fromJson(needs.generate-wheel-matrix.outputs.matrix) }} - fail-fast: false - runs-on: ${{ matrix.validation_runner }} - steps: - - name: Checkout PyTorch builder - uses: actions/checkout@v2 - - name: Validate binary wheel - uses: ./.github/actions/validate-windows-binary - with: - gpu_arch_type: ${{ matrix.gpu_arch_type }} - gpu_arch_ver: ${{ matrix.gpu_arch_version }} - installation: ${{ matrix.installation }} - python_version: ${{ matrix.python_version }} - validate-linux-libtorch-binaries: - needs: generate-libtorch-matrix + channel: ${{ inputs.channel }} + limit-win-builds: ${{ inputs.limit-win-builds }} + + win: + needs: generate-windows-matrix strategy: - matrix: - ${{ fromJson(needs.generate-libtorch-matrix.outputs.matrix) }} + matrix: ${{ fromJson(needs.generate-windows-matrix.outputs.matrix) }} fail-fast: false - runs-on: "windows-2019" - env: - PYTHON_VERSION: ${{ matrix.python_version }} - steps: - - name: Install pytorch and smoke test - shell: powershell - run: | - $install = '${{ matrix.installation }}' - Invoke-WebRequest -Uri $install -OutFile 'libtorch.zip' - Expand-Archive -Force libtorch.zip . + uses: pytorch/test-infra/.github/workflows/windows_job.yml@main + name: ${{ matrix.build_name }} + with: + runner: ${{ matrix.package_type == 'libtorch' && 'windows.4xlarge' || matrix.validation_runner }} + repository: "pytorch/builder" + ref: ${{ inputs.ref || github.ref }} + job-name: ${{ matrix.build_name }} + binary-matrix: ${{ toJSON(matrix) }} + timeout: 60 + script: | + set -ex + export ENV_NAME="conda-env-${{ github.run_id }}" + export TARGET_OS="windows" + source /c/Jenkins/Miniconda3/etc/profile.d/conda.sh + source ./.github/scripts/validate_binaries.sh diff --git a/CUDA_UPGRADE_GUIDE.MD b/CUDA_UPGRADE_GUIDE.MD index 4a725c0f06..ae3f158d31 100644 --- a/CUDA_UPGRADE_GUIDE.MD +++ b/CUDA_UPGRADE_GUIDE.MD @@ -9,9 +9,8 @@ Here is the supported matrix for CUDA and CUDNN | CUDA | CUDNN | additional details | | --- | --- | --- | -| 10.2 | 7.6.5.32 | Needed for publishing CUDA enabled binaries to PyPi since CUDA 11.x binaries don’t meet the space requirements (<750MB) | -| 11.3 | 8.3.2.44 | Stable CUDA Release | -| 11.6 | 8.3.2.44 | Latest CUDA Release | +| 11.6 | 8.3.2.44 | Stable CUDA Release | +| 11.7 | 8.5.0.96 | Latest CUDA Release | ### B. Check the package availability @@ -72,12 +71,13 @@ Add setup for our Docker `libtorch` and `manywheel`: 1. Follow this [PR 999](https://github.com/pytorch/builder/pull/999) for all steps in this section 2. To get the CUDA install link, just like with Linux, go [here](https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exe_local) and upload that `.exe` file to our S3 bucket [ossci-windows](https://s3.console.aws.amazon.com/s3/buckets/ossci-windows?region=us-east-1&tab=objects). -3. To get the cuDNN install link, you could ask NVIDIA, but you could also just sign up for an NVIDIA account and access the needed `.zip` file at this [link](https://developer.nvidia.com/rdp/cudnn-download). First click on `cuDNN Library for Windows (x86)` and then upload that zip file to our S3 bucket. -4. NOTE: When you upload files to S3, make sure to make these objects publicly readable so that our CI can access them! -5. Most times, you have to upgrade the driver install for newer versions, which would look like [updating the `windows/internal/driver_update.bat` file](https://github.com/pytorch/builder/commit/9b997037e16eb3bc635e28d101c3297d7e4ead29) +3. Review "Table 3. Possible Subpackage Names" of CUDA installation guide for windows [link](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html) to make sure the Subpackage Names have not changed. These are specified in [cuda_install.bat file](https://github.com/pytorch/builder/pull/999/files#diff-92a9c40963159c9d8f88fa2987057a65a2370737bd4ecc233498ebdfa02021e6) +4. To get the cuDNN install link, you could ask NVIDIA, but you could also just sign up for an NVIDIA account and access the needed `.zip` file at this [link](https://developer.nvidia.com/rdp/cudnn-download). First click on `cuDNN Library for Windows (x86)` and then upload that zip file to our S3 bucket. +5. NOTE: When you upload files to S3, make sure to make these objects publicly readable so that our CI can access them! +6. Most times, you have to upgrade the driver install for newer versions, which would look like [updating the `windows/internal/driver_update.bat` file](https://github.com/pytorch/builder/commit/9b997037e16eb3bc635e28d101c3297d7e4ead29) 1. Please check the CUDA Toolkit and Minimum Required Driver Version for CUDA minor version compatibility table in [the release notes](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) to see if a driver update is necessary. -6. Compile MAGMA with the new CUDA version. Update `.github/workflows/build-magma-windows.yml` to include new version. -7. Validate Magma builds by going to S3 [ossci-windows](https://s3.console.aws.amazon.com/s3/buckets/ossci-windows?region=us-east-1&tab=objects). And querying for ```magma_``` +7. Compile MAGMA with the new CUDA version. Update `.github/workflows/build-magma-windows.yml` to include new version. +8. Validate Magma builds by going to S3 [ossci-windows](https://s3.console.aws.amazon.com/s3/buckets/ossci-windows?region=us-east-1&tab=objects). And querying for ```magma_``` ## 6. Generate new Windows AMI, test and deploy to canary and prod. diff --git a/README.md b/README.md index 01e18dcbea..70d902ac32 100644 --- a/README.md +++ b/README.md @@ -10,3 +10,7 @@ Folders: - **windows** : scripts to build Windows wheels - **cron** : scripts to drive all of the above scripts across multiple configurations together - **analytics** : scripts to pull wheel download count from our AWS s3 logs + +## Testing + +In order to test build triggered by PyTorch repo's GitHub actions see [these instructions](https://github.com/pytorch/pytorch/blob/master/.github/scripts/README.md#testing-pytorchbuilder-changes) diff --git a/aarch64_linux/README.md b/aarch64_linux/README.md new file mode 100644 index 0000000000..583ed4af99 --- /dev/null +++ b/aarch64_linux/README.md @@ -0,0 +1,19 @@ +# Aarch64 (ARM/Graviton) Support Scripts +Scripts for building aarch64 PyTorch PIP Wheels. These scripts build the following wheels: +* torch +* torchvision +* torchaudio +* torchtext +* torchdata +## Aarch64_ci_build.sh +This script is design to support CD operations within PyPi manylinux aarch64 container, and be executed in the container. It prepares the container and then executes __aarch64_wheel_ci_build.py__ to build the wheels. The script "assumes" the PyTorch repo is located at: ```/pytorch``` and will put the wheels into ```/artifacts```. +### Usage +```DESIRED_PYTHON= aarch64_ci_build.sh``` + +__NOTE:__ CI build is currently __EXPERMINTAL__ + +## Build_aarch64_wheel.py +This app allows a person to build using AWS EC3 resources and requires AWS-CLI and Boto3 with AWS credentials to support building EC2 instances for the wheel builds. Can be used in a codebuild CD or from a local system. + +### Usage +```build_aarch64_wheel.py --key-name --use-docker --python 3.8 --branch ``` diff --git a/aarch64_linux/aarch64_ci_build.sh b/aarch64_linux/aarch64_ci_build.sh new file mode 100644 index 0000000000..c72698389c --- /dev/null +++ b/aarch64_linux/aarch64_ci_build.sh @@ -0,0 +1,52 @@ +#!/bin/bash +set -eux -o pipefail + +# This script is used to prepare the Docker container for aarch64_ci_wheel_build.py python script +# as we need to install conda and setup the python version for the build. + +CONDA_PYTHON_EXE=/opt/conda/bin/python +CONDA_EXE=/opt/conda/bin/conda +PATH=/opt/conda/bin:$PATH + +############################################################################### +# Install OS dependent packages +############################################################################### +yum -y install epel-release +yum -y install less zstd + +############################################################################### +# Install conda +# disable SSL_verify due to getting "Could not find a suitable TLS CA certificate bundle, invalid path" +# when using Python version, less than the conda latest +############################################################################### +echo 'Installing conda-forge' +curl -L -o /mambaforge.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh +chmod +x /mambaforge.sh +/mambaforge.sh -b -p /opt/conda +rm /mambaforge.sh +/opt/conda/bin/conda config --set ssl_verify False +/opt/conda/bin/conda install -y -c conda-forge python=${DESIRED_PYTHON} numpy pyyaml setuptools patchelf +python --version +conda --version + +############################################################################### +# Exec libglfortran.a hack +# +# libgfortran.a from quay.io/pypa/manylinux2014_aarch64 is not compiled with -fPIC. +# This causes __stack_chk_guard@@GLIBC_2.17 on pytorch build. To solve, get +# ubuntu's libgfortran.a which is compiled with -fPIC +############################################################################### +cd ~/ +curl -L -o ~/libgfortran-10-dev.deb http://ports.ubuntu.com/ubuntu-ports/pool/universe/g/gcc-10/libgfortran-10-dev_10.4.0-6ubuntu1_arm64.deb +ar x ~/libgfortran-10-dev.deb +tar --use-compress-program=unzstd -xvf data.tar.zst -C ~/ +cp -f ~/usr/lib/gcc/aarch64-linux-gnu/10/libgfortran.a /opt/rh/devtoolset-10/root/usr/lib/gcc/aarch64-redhat-linux/10/ + +############################################################################### +# Run aarch64 builder python +############################################################################### +cd / +# adding safe directory for git as the permissions will be +# on the mounted pytorch repo +git config --global --add safe.directory /pytorch +python /builder/aarch64_linux/aarch64_wheel_ci_build.py --enable-mkldnn diff --git a/aarch64_linux/aarch64_wheel_ci_build.py b/aarch64_linux/aarch64_wheel_ci_build.py new file mode 100755 index 0000000000..c76f6d6474 --- /dev/null +++ b/aarch64_linux/aarch64_wheel_ci_build.py @@ -0,0 +1,309 @@ +#!/usr/bin/env python3 + +import os +import subprocess +from typing import Dict, List, Optional, Tuple + + +'''' +Helper for getting paths for Python +''' +def list_dir(path: str) -> List[str]: + return subprocess.check_output(["ls", "-1", path]).decode().split("\n") + + +''' +Helper to get repo branches for specific versions +''' +def checkout_repo(branch: str = "main", + url: str = "", + git_clone_flags: str = "", + mapping: Dict[str, Tuple[str, str]] = []) -> Optional[str]: + for prefix in mapping: + if not branch.startswith(prefix): + continue + tag = f"v{mapping[prefix][0]}-{mapping[prefix][1]}" + os.system(f"git clone {url} -b {tag} {git_clone_flags}") + return mapping[prefix][0] + + os.system(f"git clone {url} {git_clone_flags}") + return None + + +''' +Using OpenBLAS with PyTorch +''' +def build_OpenBLAS(git_clone_flags: str = "") -> None: + print('Building OpenBLAS') + os.system(f"cd /; git clone https://github.com/xianyi/OpenBLAS -b v0.3.21 {git_clone_flags}") + make_flags = "NUM_THREADS=64 USE_OPENMP=1 NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=ARMV8 " + os.system(f"cd OpenBLAS; make {make_flags} -j8; make {make_flags} install; cd /; rm -rf OpenBLAS") + + +''' +Using ArmComputeLibrary for aarch64 PyTorch +''' +def build_ArmComputeLibrary(git_clone_flags: str = "") -> None: + print('Building Arm Compute Library') + os.system("cd / && mkdir /acl") + os.system(f"git clone https://github.com/ARM-software/ComputeLibrary.git -b v22.11 {git_clone_flags}") + os.system(f"cd ComputeLibrary; export acl_install_dir=/acl; " \ + f"scons Werror=1 -j8 debug=0 neon=1 opencl=0 os=linux openmp=1 cppthreads=0 arch=armv8.2-a multi_isa=1 build=native build_dir=$acl_install_dir/build; " \ + f"cp -r arm_compute $acl_install_dir; " \ + f"cp -r include $acl_install_dir; " \ + f"cp -r utils $acl_install_dir; " \ + f"cp -r support $acl_install_dir; " \ + f"cp -r src $acl_install_dir; cd /") + + +''' +Script to embed libgomp to the wheels +''' +def embed_libgomp(wheel_name) -> None: + print('Embedding libgomp into wheel') + os.system(f"python3 /builder/aarch64_linux/embed_library.py {wheel_name} --update-tag") + + +''' +Build TorchVision wheel +''' +def build_torchvision(branch: str = "main", + git_clone_flags: str = "") -> str: + print('Checking out TorchVision repo') + build_version = checkout_repo(branch=branch, + url="https://github.com/pytorch/vision", + git_clone_flags=git_clone_flags, + mapping={ + "v1.7.1": ("0.8.2", "rc2"), + "v1.8.0": ("0.9.0", "rc3"), + "v1.8.1": ("0.9.1", "rc1"), + "v1.9.0": ("0.10.0", "rc1"), + "v1.10.0": ("0.11.1", "rc1"), + "v1.10.1": ("0.11.2", "rc1"), + "v1.10.2": ("0.11.3", "rc1"), + "v1.11.0": ("0.12.0", "rc1"), + "v1.12.0": ("0.13.0", "rc4"), + "v1.12.1": ("0.13.1", "rc6"), + "v1.13.0": ("0.14.0", "rc4"), + "v1.13.1": ("0.14.1", "rc2"), + "v2.0.0": ("0.15.0", "rc2"), + }) + print('Building TorchVision wheel') + build_vars = "CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000 " + if branch == 'nightly': + version = '' + if os.path.exists('/vision/version.txt'): + version = subprocess.check_output(['cat', '/vision/version.txt']).decode().strip() + if len(version) == 0: + # In older revisions, version was embedded in setup.py + version = subprocess.check_output(['grep', 'version', 'setup.py']).decode().strip().split('\'')[1][:-2] + build_date = subprocess.check_output(['git','log','--pretty=format:%cs','-1'], cwd='/vision').decode().replace('-','') + build_vars += f"BUILD_VERSION={version}.dev{build_date}" + elif build_version is not None: + build_vars += f"BUILD_VERSION={build_version}" + + os.system(f"cd /vision; {build_vars} python3 setup.py bdist_wheel") + wheel_name = list_dir("/vision/dist")[0] + embed_libgomp(f"/vision/dist/{wheel_name}") + + print('Move TorchVision wheel to artfacts') + os.system(f"mv /vision/dist/{wheel_name} /artifacts/") + return wheel_name + + +''' +Build TorchAudio wheel +''' +def build_torchaudio(branch: str = "main", + git_clone_flags: str = "") -> str: + print('Checking out TorchAudio repo') + git_clone_flags += " --recurse-submodules" + build_version = checkout_repo(branch=branch, + url="https://github.com/pytorch/audio", + git_clone_flags=git_clone_flags, + mapping={ + "v1.9.0": ("0.9.0", "rc2"), + "v1.10.0": ("0.10.0", "rc5"), + "v1.10.1": ("0.10.1", "rc1"), + "v1.10.2": ("0.10.2", "rc1"), + "v1.11.0": ("0.11.0", "rc1"), + "v1.12.0": ("0.12.0", "rc3"), + "v1.12.1": ("0.12.1", "rc5"), + "v1.13.0": ("0.13.0", "rc4"), + "v1.13.1": ("0.13.1", "rc2"), + "v2.0.0": ("2.0.0", "rc2"), + }) + print('Building TorchAudio wheel') + build_vars = "CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000 " + if branch == 'nightly': + version = '' + if os.path.exists('/audio/version.txt'): + version = subprocess.check_output(['cat', '/audio/version.txt']).decode().strip() + build_date = subprocess.check_output(['git','log','--pretty=format:%cs','-1'], cwd='/audio').decode().replace('-','') + build_vars += f"BUILD_VERSION={version}.dev{build_date}" + elif build_version is not None: + build_vars += f"BUILD_VERSION={build_version}" + + os.system(f"cd /audio; {build_vars} python3 setup.py bdist_wheel") + wheel_name = list_dir("/audio/dist")[0] + embed_libgomp(f"/audio/dist/{wheel_name}") + + print('Move TorchAudio wheel to artfacts') + os.system(f"mv /audio/dist/{wheel_name} /artifacts/") + return wheel_name + + +''' +Build TorchText wheel +''' +def build_torchtext(branch: str = "main", + git_clone_flags: str = "") -> str: + print('Checking out TorchText repo') + os.system(f"cd /") + git_clone_flags += " --recurse-submodules" + build_version = checkout_repo(branch=branch, + url="https://github.com/pytorch/text", + git_clone_flags=git_clone_flags, + mapping={ + "v1.9.0": ("0.10.0", "rc1"), + "v1.10.0": ("0.11.0", "rc2"), + "v1.10.1": ("0.11.1", "rc1"), + "v1.10.2": ("0.11.2", "rc1"), + "v1.11.0": ("0.12.0", "rc1"), + "v1.12.0": ("0.13.0", "rc2"), + "v1.12.1": ("0.13.1", "rc5"), + "v1.13.0": ("0.14.0", "rc3"), + "v1.13.1": ("0.14.1", "rc1"), + "v2.0.0": ("0.15.0", "rc2"), + }) + print('Building TorchText wheel') + build_vars = "CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000 " + if branch == 'nightly': + version = '' + if os.path.exists('/text/version.txt'): + version = subprocess.check_output(['cat', '/text/version.txt']).decode().strip() + build_date = subprocess.check_output(['git','log','--pretty=format:%cs','-1'], cwd='/text').decode().replace('-','') + build_vars += f"BUILD_VERSION={version}.dev{build_date}" + elif build_version is not None: + build_vars += f"BUILD_VERSION={build_version}" + + os.system(f"cd text; {build_vars} python3 setup.py bdist_wheel") + wheel_name = list_dir("/text/dist")[0] + embed_libgomp(f"/text/dist/{wheel_name}") + + print('Move TorchText wheel to artfacts') + os.system(f"mv /text/dist/{wheel_name} /artifacts/") + return wheel_name + + +''' +Build TorchData wheel +''' +def build_torchdata(branch: str = "main", + git_clone_flags: str = "") -> str: + print('Checking out TorchData repo') + git_clone_flags += " --recurse-submodules" + build_version = checkout_repo(branch=branch, + url="https://github.com/pytorch/data", + git_clone_flags=git_clone_flags, + mapping={ + "v1.11.0": ("0.3.0", "rc1"), + "v1.12.0": ("0.4.0", "rc3"), + "v1.12.1": ("0.4.1", "rc5"), + "v1.13.1": ("0.5.1", "rc2"), + "v2.0.0": ("0.6.0", "rc2"), + }) + print('Building TorchData wheel') + build_vars = "CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000 " + if branch == 'nightly': + version = '' + if os.path.exists('/data/version.txt'): + version = subprocess.check_output(['cat', '/data/version.txt']).decode().strip() + build_date = subprocess.check_output(['git','log','--pretty=format:%cs','-1'], cwd='/data').decode().replace('-','') + build_vars += f"BUILD_VERSION={version}.dev{build_date}" + elif build_version is not None: + build_vars += f"BUILD_VERSION={build_version}" + + os.system(f"cd /data; {build_vars} python3 setup.py bdist_wheel") + wheel_name = list_dir("/data/dist")[0] + embed_libgomp(f"/data/dist/{wheel_name}") + + print('Move TorchAudio wheel to artfacts') + os.system(f"mv /data/dist/{wheel_name} /artifacts/") + return wheel_name + + +def parse_arguments(): + from argparse import ArgumentParser + parser = ArgumentParser("AARCH64 wheels python CD") + parser.add_argument("--debug", action="store_true") + parser.add_argument("--build-only", action="store_true") + parser.add_argument("--test-only", type=str) + parser.add_argument("--enable-mkldnn", action="store_true") + return parser.parse_args() + + +''' +Entry Point +''' +if __name__ == '__main__': + + args = parse_arguments() + enable_mkldnn = args.enable_mkldnn + os.system("cd /pytorch") + branch = subprocess.check_output("git rev-parse --abbrev-ref HEAD") + + git_clone_flags = " --depth 1 --shallow-submodules" + os.system(f"conda install -y ninja scons") + + print("Build and Install OpenBLAS") + build_OpenBLAS(git_clone_flags) + + print('Building PyTorch wheel') + build_vars = "CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000 " + os.system(f"cd /pytorch; pip install -r requirements.txt") + os.system(f"pip install auditwheel") + os.system(f"python setup.py clean") + + if branch == 'nightly' or branch == 'master': + build_date = subprocess.check_output(['git','log','--pretty=format:%cs','-1'], cwd='/pytorch').decode().replace('-','') + version = subprocess.check_output(['cat','version.txt'], cwd='/pytorch').decode().strip()[:-2] + build_vars += f"BUILD_TEST=0 PYTORCH_BUILD_VERSION={version}.dev{build_date} PYTORCH_BUILD_NUMBER=1" + if branch.startswith("v1.") or branch.startswith("v2."): + build_vars += f"BUILD_TEST=0 PYTORCH_BUILD_VERSION={branch[1:branch.find('-')]} PYTORCH_BUILD_NUMBER=1" + if enable_mkldnn: + build_ArmComputeLibrary(git_clone_flags) + print("build pytorch with mkldnn+acl backend") + os.system(f"export ACL_ROOT_DIR=/acl; export LD_LIBRARY_PATH=/acl/build; export ACL_LIBRARY=/acl/build") + build_vars += " USE_MKLDNN=ON USE_MKLDNN_ACL=ON" + os.system(f"cd /pytorch; {build_vars} python3 setup.py bdist_wheel") + print('Repair the wheel') + pytorch_wheel_name = list_dir("pytorch/dist")[0] + os.system(f"export LD_LIBRARY_PATH=/pytorch/build/lib:$LD_LIBRARY_PATH; auditwheel repair /pytorch/dist/{pytorch_wheel_name}") + print('replace the original wheel with the repaired one') + pytorch_repaired_wheel_name = list_dir("wheelhouse")[0] + os.system(f"cp /wheelhouse/{pytorch_repaired_wheel_name} /pytorch/dist/{pytorch_wheel_name}") + else: + print("build pytorch without mkldnn backend") + os.system(f"cd pytorch ; {build_vars} python3 setup.py bdist_wheel") + + print("Deleting build folder") + os.system("cd /pytorch; rm -rf build") + pytorch_wheel_name = list_dir("/pytorch/dist")[0] + embed_libgomp(f"/pytorch/dist/{pytorch_wheel_name}") + print('Move PyTorch wheel to artfacts') + os.system(f"mv /pytorch/dist/{pytorch_wheel_name} /artifacts/") + print("Installing Pytorch wheel") + os.system(f"pip install /artifacts/{pytorch_wheel_name}") + + vision_wheel_name = build_torchvision(branch=branch, git_clone_flags=git_clone_flags) + audio_wheel_name = build_torchaudio(branch=branch, git_clone_flags=git_clone_flags) + text_wheel_name = build_torchtext(branch=branch, git_clone_flags=git_clone_flags) + data_wheel_name = build_torchdata(branch=branch, git_clone_flags=git_clone_flags) + + print(f"Wheels Created:\n" \ + f"{pytorch_wheel_name}\n" \ + f"{vision_wheel_name}\n" \ + f"{audio_wheel_name}\n" \ + f"{text_wheel_name}\n" \ + f"{data_wheel_name}\n") diff --git a/build_aarch64_wheel.py b/aarch64_linux/build_aarch64_wheel.py similarity index 74% rename from build_aarch64_wheel.py rename to aarch64_linux/build_aarch64_wheel.py index ee3fa54d10..f7b70208a2 100755 --- a/build_aarch64_wheel.py +++ b/aarch64_linux/build_aarch64_wheel.py @@ -4,7 +4,7 @@ # To generate binaries for the release follow these steps: # 1. Update mappings for each of the Domain Libraries by adding new row to a table like this: "v1.11.0": ("0.11.0", "rc1"), # 2. Run script with following arguments for each of the supported python versions and specify required RC tag for example: v1.11.0-rc3: -# build_aarch64_wheel.py --key-name --use-docker --python 3.7 --branch +# build_aarch64_wheel.py --key-name --use-docker --python 3.8 --branch import boto3 @@ -15,11 +15,11 @@ from typing import Dict, List, Optional, Tuple, Union - # AMI images for us-east-1, change the following based on your ~/.aws/config os_amis = { - 'ubuntu18_04': "ami-0f2b111fdc1647918", # login_name: ubuntu - 'ubuntu20_04': "ami-0ea142bd244023692", # login_name: ubuntu + 'ubuntu18_04': "ami-078eece1d8119409f", # login_name: ubuntu + 'ubuntu20_04': "ami-052eac90edaa9d08f", # login_name: ubuntu + 'ubuntu22_04': "ami-0c6c29c5125214c77", # login_name: ubuntu 'redhat8': "ami-0698b90665a2ddcf1", # login_name: ec2-user } ubuntu18_04_ami = os_amis['ubuntu18_04'] @@ -128,7 +128,7 @@ def run_cmd(self, args: Union[str, List[str]]) -> None: assert self.container_id is not None docker_cmd = self._gen_ssh_prefix() + ['docker', 'exec', '-i', self.container_id, 'bash'] p = subprocess.Popen(docker_cmd, stdin=subprocess.PIPE) - p.communicate(input=" ".join(["source .bashrc;"] + self._split_cmd(args)).encode("utf-8")) + p.communicate(input=" ".join(["source .bashrc && "] + self._split_cmd(args)).encode("utf-8")) rc = p.wait() if rc != 0: raise subprocess.CalledProcessError(rc, docker_cmd) @@ -139,7 +139,7 @@ def check_output(self, args: Union[str, List[str]]) -> str: assert self.container_id is not None docker_cmd = self._gen_ssh_prefix() + ['docker', 'exec', '-i', self.container_id, 'bash'] p = subprocess.Popen(docker_cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE) - (out, err) = p.communicate(input=" ".join(["source .bashrc;"] + self._split_cmd(args)).encode("utf-8")) + (out, err) = p.communicate(input=" ".join(["source .bashrc && "] + self._split_cmd(args)).encode("utf-8")) rc = p.wait() if rc != 0: raise subprocess.CalledProcessError(rc, docker_cmd, output=out, stderr=err) @@ -211,8 +211,12 @@ def install_condaforge_python(host: RemoteHost, python_version="3.8") -> None: # Python-3.6 EOLed and not compatible with conda-4.11 install_condaforge(host, suffix="download/4.10.3-10/Miniforge3-4.10.3-10-Linux-aarch64.sh") host.run_cmd(f"conda install -y python={python_version} numpy pyyaml") + elif python_version == "3.11": + install_condaforge(host, suffix="download/4.11.0-4/Miniforge3-4.11.0-4-Linux-aarch64.sh") + # Pytorch-1.10 or older are not compatible with setuptools=59.6 or newer + host.run_cmd(f"conda install -y python={python_version} numpy pyyaml setuptools=59.8.0 -c malfet") else: - install_condaforge(host) + install_condaforge(host, suffix="download/4.11.0-4/Miniforge3-4.11.0-4-Linux-aarch64.sh") # Pytorch-1.10 or older are not compatible with setuptools=59.6 or newer host.run_cmd(f"conda install -y python={python_version} numpy pyyaml setuptools=59.5.0") @@ -221,16 +225,16 @@ def build_OpenBLAS(host: RemoteHost, git_clone_flags: str = "") -> None: print('Building OpenBLAS') host.run_cmd(f"git clone https://github.com/xianyi/OpenBLAS -b v0.3.19 {git_clone_flags}") make_flags = "NUM_THREADS=64 USE_OPENMP=1 NO_SHARED=1 DYNAMIC_ARCH=1 TARGET=ARMV8" - host.run_cmd(f"pushd OpenBLAS; make {make_flags} -j8; sudo make {make_flags} install; popd; rm -rf OpenBLAS") + host.run_cmd(f"pushd OpenBLAS && make {make_flags} -j8 && sudo make {make_flags} install && popd && rm -rf OpenBLAS") -def build_FFTW(host: RemoteHost, git_clone_flags: str = "") -> None: - print("Building FFTW3") - host.run_cmd("sudo apt-get install -y ocaml ocamlbuild autoconf automake indent libtool fig2dev texinfo") - # TODO: fix a version to build - # TODO: consider adding flags --host=arm-linux-gnueabi --enable-single --enable-neon CC=arm-linux-gnueabi-gcc -march=armv7-a -mfloat-abi=softfp - host.run_cmd(f"git clone https://github.com/FFTW/fftw3 {git_clone_flags}") - host.run_cmd("pushd fftw3; sh bootstrap.sh; make -j8; sudo make install; popd") +def build_ArmComputeLibrary(host: RemoteHost, git_clone_flags: str = "") -> None: + print('Building Arm Compute Library') + acl_install_dir="${HOME}/acl" + acl_build_flags="debug=0 neon=1 opencl=0 os=linux openmp=1 cppthreads=0 arch=armv8.2-a multi_isa=1 build=native" + host.run_cmd(f"mkdir {acl_install_dir}") + host.run_cmd(f"git clone https://github.com/ARM-software/ComputeLibrary.git -b v22.11 {git_clone_flags}") + host.run_cmd(f"cd ComputeLibrary && scons Werror=1 -j8 {acl_build_flags} build_dir={acl_install_dir}/build") def embed_libgomp(host: RemoteHost, use_conda, wheel_name) -> None: @@ -250,7 +254,7 @@ def embed_libgomp(host: RemoteHost, use_conda, wheel_name) -> None: def checkout_repo(host: RemoteHost, *, - branch: str = "master", + branch: str = "main", url: str, git_clone_flags: str, mapping: Dict[str, Tuple[str, str]]) -> Optional[str]: @@ -261,14 +265,19 @@ def checkout_repo(host: RemoteHost, *, host.run_cmd(f"git clone {url} -b {tag} {git_clone_flags}") return mapping[prefix][0] - host.run_cmd(f"git clone {url} {git_clone_flags}") + # Map master to main + if branch == "master" and url.rsplit("/")[-1] in ['vision', 'text', 'audio', 'data']: + branch = "main" + + host.run_cmd(f"git clone {url} -b {branch} {git_clone_flags}") return None def build_torchvision(host: RemoteHost, *, - branch: str = "master", + branch: str = "main", use_conda: bool = True, - git_clone_flags: str) -> str: + git_clone_flags: str, + run_smoke_tests: bool = True) -> str: print('Checking out TorchVision repo') build_version = checkout_repo(host, branch=branch, @@ -284,33 +293,84 @@ def build_torchvision(host: RemoteHost, *, "v1.10.2": ("0.11.3", "rc1"), "v1.11.0": ("0.12.0", "rc1"), "v1.12.0": ("0.13.0", "rc4"), + "v1.12.1": ("0.13.1", "rc6"), + "v1.13.0": ("0.14.0", "rc4"), + "v1.13.1": ("0.14.1", "rc2"), + "v2.0.0": ("0.15.1", "rc2"), }) - print('Building TorchVision wheel') + print("Building TorchVision wheel") + + # Please note libnpg and jpeg are required to build image.so extension + if use_conda: + host.run_cmd("conda install -y libpng jpeg") + # Remove .so files to force static linking + host.run_cmd("rm miniforge3/lib/libpng.so miniforge3/lib/libpng16.so miniforge3/lib/libjpeg.so") + # And patch setup.py to include libz dependency for libpng + host.run_cmd(['sed -i -e \'s/image_link_flags\.append("png")/image_link_flags += ["png", "z"]/\' vision/setup.py']) + build_vars = "" - if branch == 'nightly': + if branch == "nightly": version = host.check_output(["if [ -f vision/version.txt ]; then cat vision/version.txt; fi"]).strip() if len(version) == 0: # In older revisions, version was embedded in setup.py version = host.check_output(["grep", "\"version = '\"", "vision/setup.py"]).strip().split("'")[1][:-2] - build_date = host.check_output("cd pytorch ; git log --pretty=format:%s -1").strip().split()[0].replace("-", "") + build_date = host.check_output("cd vision && git log --pretty=format:%s -1").strip().split()[0].replace("-", "") build_vars += f"BUILD_VERSION={version}.dev{build_date}" elif build_version is not None: - build_vars += f"BUILD_VERSION={build_version}" + build_vars += f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-')[0]}" if host.using_docker(): build_vars += " CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000" - host.run_cmd(f"cd vision; {build_vars} python3 setup.py bdist_wheel") + host.run_cmd(f"cd vision && {build_vars} python3 setup.py bdist_wheel") vision_wheel_name = host.list_dir("vision/dist")[0] embed_libgomp(host, use_conda, os.path.join('vision', 'dist', vision_wheel_name)) print('Copying TorchVision wheel') host.download_wheel(os.path.join('vision', 'dist', vision_wheel_name)) + if run_smoke_tests: + host.run_cmd(f"pip3 install {os.path.join('vision', 'dist', vision_wheel_name)}") + host.run_cmd("python3 vision/test/smoke_test.py") print("Delete vision checkout") host.run_cmd("rm -rf vision") return vision_wheel_name +def build_torchdata(host: RemoteHost, *, + branch: str = "master", + use_conda: bool = True, + git_clone_flags: str = "") -> str: + print('Checking out TorchData repo') + git_clone_flags += " --recurse-submodules" + build_version = checkout_repo(host, + branch=branch, + url="https://github.com/pytorch/data", + git_clone_flags=git_clone_flags, + mapping={ + "v1.13.1": ("0.5.1", ""), + "v2.0.0": ("0.6.0", "rc5"), + }) + print('Building TorchData wheel') + build_vars = "" + if branch == 'nightly': + version = host.check_output(["if [ -f data/version.txt ]; then cat data/version.txt; fi"]).strip() + build_date = host.check_output("cd data && git log --pretty=format:%s -1").strip().split()[0].replace("-", "") + build_vars += f"BUILD_VERSION={version}.dev{build_date}" + elif build_version is not None: + build_vars += f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-')[0]}" + if host.using_docker(): + build_vars += " CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000" + + host.run_cmd(f"cd data && {build_vars} python3 setup.py bdist_wheel") + wheel_name = host.list_dir("data/dist")[0] + embed_libgomp(host, use_conda, os.path.join('data', 'dist', wheel_name)) + + print('Copying TorchData wheel') + host.download_wheel(os.path.join('data', 'dist', wheel_name)) + + return wheel_name + + def build_torchtext(host: RemoteHost, *, branch: str = "master", use_conda: bool = True, @@ -328,19 +388,23 @@ def build_torchtext(host: RemoteHost, *, "v1.10.2": ("0.11.2", "rc1"), "v1.11.0": ("0.12.0", "rc1"), "v1.12.0": ("0.13.0", "rc2"), + "v1.12.1": ("0.13.1", "rc5"), + "v1.13.0": ("0.14.0", "rc3"), + "v1.13.1": ("0.14.1", "rc1"), + "v2.0.0": ("0.15.1", "rc2"), }) print('Building TorchText wheel') build_vars = "" if branch == 'nightly': version = host.check_output(["if [ -f text/version.txt ]; then cat text/version.txt; fi"]).strip() - build_date = host.check_output("cd pytorch ; git log --pretty=format:%s -1").strip().split()[0].replace("-", "") + build_date = host.check_output("cd text && git log --pretty=format:%s -1").strip().split()[0].replace("-", "") build_vars += f"BUILD_VERSION={version}.dev{build_date}" elif build_version is not None: - build_vars += f"BUILD_VERSION={build_version}" + build_vars += f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-')[0]}" if host.using_docker(): build_vars += " CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000" - host.run_cmd(f"cd text; {build_vars} python3 setup.py bdist_wheel") + host.run_cmd(f"cd text && {build_vars} python3 setup.py bdist_wheel") wheel_name = host.list_dir("text/dist")[0] embed_libgomp(host, use_conda, os.path.join('text', 'dist', wheel_name)) @@ -367,19 +431,23 @@ def build_torchaudio(host: RemoteHost, *, "v1.10.2": ("0.10.2", "rc1"), "v1.11.0": ("0.11.0", "rc1"), "v1.12.0": ("0.12.0", "rc3"), + "v1.12.1": ("0.12.1", "rc5"), + "v1.13.0": ("0.13.0", "rc4"), + "v1.13.1": ("0.13.1", "rc2"), + "v2.0.0": ("2.0.1", "rc3"), }) print('Building TorchAudio wheel') build_vars = "" if branch == 'nightly': version = host.check_output(["grep", "\"version = '\"", "audio/setup.py"]).strip().split("'")[1][:-2] - build_date = host.check_output("cd pytorch ; git log --pretty=format:%s -1").strip().split()[0].replace("-", "") + build_date = host.check_output("cd audio && git log --pretty=format:%s -1").strip().split()[0].replace("-", "") build_vars += f"BUILD_VERSION={version}.dev{build_date}" elif build_version is not None: - build_vars += f"BUILD_VERSION={build_version}" + build_vars += f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-')[0]}" if host.using_docker(): build_vars += " CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000" - host.run_cmd(f"cd audio; {build_vars} python3 setup.py bdist_wheel") + host.run_cmd(f"cd audio && {build_vars} python3 setup.py bdist_wheel") wheel_name = host.list_dir("audio/dist")[0] embed_libgomp(host, use_conda, os.path.join('audio', 'dist', wheel_name)) @@ -390,9 +458,9 @@ def build_torchaudio(host: RemoteHost, *, def configure_system(host: RemoteHost, *, - compiler="gcc-8", - use_conda=True, - python_version="3.8") -> None: + compiler: str = "gcc-8", + use_conda: bool = True, + python_version: str = "3.8") -> None: if use_conda: install_condaforge_python(host, python_version) @@ -402,7 +470,7 @@ def configure_system(host: RemoteHost, *, host.run_cmd("sudo apt-get install -y ninja-build g++ git cmake gfortran unzip") else: host.run_cmd("yum install -y sudo") - host.run_cmd("conda install -y ninja") + host.run_cmd("conda install -y ninja scons") if not use_conda: host.run_cmd("sudo apt-get install -y python3-dev python3-yaml python3-setuptools python3-wheel python3-pip") @@ -419,23 +487,39 @@ def configure_system(host: RemoteHost, *, host.run_cmd("sudo pip3 install numpy") +def build_domains(host: RemoteHost, *, + branch: str = "master", + use_conda: bool = True, + git_clone_flags: str = "") -> Tuple[str, str, str, str]: + vision_wheel_name = build_torchvision(host, branch=branch, use_conda=use_conda, git_clone_flags=git_clone_flags) + audio_wheel_name = build_torchaudio(host, branch=branch, use_conda=use_conda, git_clone_flags=git_clone_flags) + data_wheel_name = build_torchdata(host, branch=branch, use_conda=use_conda, git_clone_flags=git_clone_flags) + text_wheel_name = build_torchtext(host, branch=branch, use_conda=use_conda, git_clone_flags=git_clone_flags) + return (vision_wheel_name, audio_wheel_name, data_wheel_name, text_wheel_name) + + def start_build(host: RemoteHost, *, - branch="master", - compiler="gcc-8", - use_conda=True, - python_version="3.8", - shallow_clone=True) -> Tuple[str, str]: + branch: str = "master", + compiler: str = "gcc-8", + use_conda: bool = True, + python_version: str = "3.8", + pytorch_only: bool = False, + pytorch_build_number: Optional[str] = None, + shallow_clone: bool = True, + enable_mkldnn: bool = False) -> Tuple[str, str, str, str, str]: git_clone_flags = " --depth 1 --shallow-submodules" if shallow_clone else "" if host.using_docker() and not use_conda: print("Auto-selecting conda option for docker images") use_conda = True + if not host.using_docker(): + print("Disable mkldnn for host builds") + enable_mkldnn = False configure_system(host, compiler=compiler, use_conda=use_conda, python_version=python_version) build_OpenBLAS(host, git_clone_flags) - # build_FFTW(host, git_clone_flags) if host.using_docker(): print("Move libgfortant.a into a standard location") @@ -452,19 +536,36 @@ def start_build(host: RemoteHost, *, host.run_cmd(f"git clone --recurse-submodules -b {branch} https://github.com/pytorch/pytorch {git_clone_flags}") print('Building PyTorch wheel') + build_opts = "" + if pytorch_build_number is not None: + build_opts += f" --build-number {pytorch_build_number}" # Breakpad build fails on aarch64 build_vars = "USE_BREAKPAD=0 " if branch == 'nightly': - build_date = host.check_output("cd pytorch ; git log --pretty=format:%s -1").strip().split()[0].replace("-", "") + build_date = host.check_output("cd pytorch && git log --pretty=format:%s -1").strip().split()[0].replace("-", "") version = host.check_output("cat pytorch/version.txt").strip()[:-2] build_vars += f"BUILD_TEST=0 PYTORCH_BUILD_VERSION={version}.dev{build_date} PYTORCH_BUILD_NUMBER=1" - if branch.startswith("v1."): + if branch.startswith("v1.") or branch.startswith("v2."): build_vars += f"BUILD_TEST=0 PYTORCH_BUILD_VERSION={branch[1:branch.find('-')]} PYTORCH_BUILD_NUMBER=1" if host.using_docker(): build_vars += " CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000" - host.run_cmd(f"cd pytorch ; {build_vars} python3 setup.py bdist_wheel") + if enable_mkldnn: + build_ArmComputeLibrary(host, git_clone_flags) + print("build pytorch with mkldnn+acl backend") + build_vars += " USE_MKLDNN=ON USE_MKLDNN_ACL=ON" + host.run_cmd(f"cd pytorch && export ACL_ROOT_DIR=$HOME/ComputeLibrary:$HOME/acl && {build_vars} python3 setup.py bdist_wheel{build_opts}") + print('Repair the wheel') + pytorch_wheel_name = host.list_dir("pytorch/dist")[0] + host.run_cmd(f"export LD_LIBRARY_PATH=$HOME/acl/build:$HOME/pytorch/build/lib && auditwheel repair $HOME/pytorch/dist/{pytorch_wheel_name}") + print('replace the original wheel with the repaired one') + pytorch_repaired_wheel_name = host.list_dir("wheelhouse")[0] + host.run_cmd(f"cp $HOME/wheelhouse/{pytorch_repaired_wheel_name} $HOME/pytorch/dist/{pytorch_wheel_name}") + else: + print("build pytorch without mkldnn backend") + host.run_cmd(f"cd pytorch && {build_vars} python3 setup.py bdist_wheel{build_opts}") + print("Deleting build folder") - host.run_cmd("cd pytorch; rm -rf build") + host.run_cmd("cd pytorch && rm -rf build") pytorch_wheel_name = host.list_dir("pytorch/dist")[0] embed_libgomp(host, use_conda, os.path.join('pytorch', 'dist', pytorch_wheel_name)) print('Copying the wheel') @@ -473,11 +574,11 @@ def start_build(host: RemoteHost, *, print('Installing PyTorch wheel') host.run_cmd(f"pip3 install pytorch/dist/{pytorch_wheel_name}") - vision_wheel_name = build_torchvision(host, branch=branch, use_conda=use_conda, git_clone_flags=git_clone_flags) - build_torchaudio(host, branch=branch, use_conda=use_conda, git_clone_flags=git_clone_flags) - build_torchtext(host, branch=branch, use_conda=use_conda, git_clone_flags=git_clone_flags) + if pytorch_only: + return (pytorch_wheel_name, None, None, None, None) + domain_wheels = build_domains(host, branch=branch, use_conda=use_conda, git_clone_flags=git_clone_flags) - return pytorch_wheel_name, vision_wheel_name + return (pytorch_wheel_name, *domain_wheels) embed_library_script = """ @@ -602,10 +703,11 @@ def parse_arguments(): parser.add_argument("--debug", action="store_true") parser.add_argument("--build-only", action="store_true") parser.add_argument("--test-only", type=str) - parser.add_argument("--os", type=str, choices=list(os_amis.keys()), default='ubuntu18_04') - parser.add_argument("--python-version", type=str, choices=['3.6', '3.7', '3.8', '3.9', '3.10'], default=None) + parser.add_argument("--os", type=str, choices=list(os_amis.keys()), default='ubuntu20_04') + parser.add_argument("--python-version", type=str, choices=['3.6', '3.7', '3.8', '3.9', '3.10', '3.11'], default=None) parser.add_argument("--alloc-instance", action="store_true") parser.add_argument("--list-instances", action="store_true") + parser.add_argument("--pytorch-only", action="store_true") parser.add_argument("--keep-running", action="store_true") parser.add_argument("--terminate-instances", action="store_true") parser.add_argument("--instance-type", type=str, default="t4g.2xlarge") @@ -613,6 +715,8 @@ def parse_arguments(): parser.add_argument("--use-docker", action="store_true") parser.add_argument("--compiler", type=str, choices=['gcc-7', 'gcc-8', 'gcc-9', 'clang'], default="gcc-8") parser.add_argument("--use-torch-from-pypi", action="store_true") + parser.add_argument("--pytorch-build-number", type=str, default=None) + parser.add_argument("--disable-mkldnn", action="store_true") return parser.parse_args() @@ -639,7 +743,7 @@ def parse_arguments(): check `~/.ssh/` folder or manually set SSH_KEY_PATH environment variable.""") # Starting the instance - inst = start_instance(key_name, ami=ami) + inst = start_instance(key_name, ami=ami, instance_type=args.instance_type) instance_name = f'{args.key_name}-{args.os}' if args.python_version is not None: instance_name += f'-py{args.python_version}' @@ -673,14 +777,17 @@ def parse_arguments(): python_version=python_version) print("Installing PyTorch wheel") host.run_cmd("pip3 install torch") - build_torchvision(host, - branch=args.branch, - git_clone_flags=" --depth 1 --shallow-submodules") + build_domains(host, + branch=args.branch, + git_clone_flags=" --depth 1 --shallow-submodules") else: start_build(host, branch=args.branch, compiler=args.compiler, - python_version=python_version) + python_version=python_version, + pytorch_only=args.pytorch_only, + pytorch_build_number=args.pytorch_build_number, + enable_mkldnn=not args.disable_mkldnn) if not args.keep_running: print(f'Waiting for instance {inst.id} to terminate') inst.terminate() diff --git a/aarch64_linux/embed_library.py b/aarch64_linux/embed_library.py new file mode 100644 index 0000000000..978970d45f --- /dev/null +++ b/aarch64_linux/embed_library.py @@ -0,0 +1,72 @@ +#!/usr/bin/env python3 + +from auditwheel.patcher import Patchelf +from auditwheel.wheeltools import InWheelCtx +from auditwheel.elfutils import elf_file_filter +from auditwheel.repair import copylib +from auditwheel.lddtree import lddtree +from subprocess import check_call +import os +import shutil +import sys +from tempfile import TemporaryDirectory + + +def replace_tag(filename): + with open(filename, 'r') as f: + lines = f.read().split("\\n") + for i,line in enumerate(lines): + if not line.startswith("Tag: "): + continue + lines[i] = line.replace("-linux_", "-manylinux2014_") + print(f'Updated tag from {line} to {lines[i]}') + + with open(filename, 'w') as f: + f.write("\\n".join(lines)) + + +class AlignedPatchelf(Patchelf): + def set_soname(self, file_name: str, new_soname: str) -> None: + check_call(['patchelf', '--page-size', '65536', '--set-soname', new_soname, file_name]) + + def replace_needed(self, file_name: str, soname: str, new_soname: str) -> None: + check_call(['patchelf', '--page-size', '65536', '--replace-needed', soname, new_soname, file_name]) + + +def embed_library(whl_path, lib_soname, update_tag=False): + patcher = AlignedPatchelf() + out_dir = TemporaryDirectory() + whl_name = os.path.basename(whl_path) + tmp_whl_name = os.path.join(out_dir.name, whl_name) + with InWheelCtx(whl_path) as ctx: + torchlib_path = os.path.join(ctx._tmpdir.name, 'torch', 'lib') + ctx.out_wheel=tmp_whl_name + new_lib_path, new_lib_soname = None, None + for filename, elf in elf_file_filter(ctx.iter_files()): + if not filename.startswith('torch/lib'): + continue + libtree = lddtree(filename) + if lib_soname not in libtree['needed']: + continue + lib_path = libtree['libs'][lib_soname]['path'] + if lib_path is None: + print(f"Can't embed {lib_soname} as it could not be found") + break + if lib_path.startswith(torchlib_path): + continue + + if new_lib_path is None: + new_lib_soname, new_lib_path = copylib(lib_path, torchlib_path, patcher) + patcher.replace_needed(filename, lib_soname, new_lib_soname) + print(f'Replacing {lib_soname} with {new_lib_soname} for {filename}') + if update_tag: + # Add manylinux2014 tag + for filename in ctx.iter_files(): + if os.path.basename(filename) != 'WHEEL': + continue + replace_tag(filename) + shutil.move(tmp_whl_name, whl_path) + + +if __name__ == '__main__': + embed_library(sys.argv[1], 'libgomp.so.1', len(sys.argv) > 2 and sys.argv[2] == '--update-tag') diff --git a/analytics/github_analyze.py b/analytics/github_analyze.py index c255c6c8f1..47330208c2 100755 --- a/analytics/github_analyze.py +++ b/analytics/github_analyze.py @@ -161,9 +161,12 @@ def __init__(self, path, remote='upstream'): self.repo_dir = path self.remote = remote + def _run_git_cmd(self, *args) -> str: + return _check_output(['git', '-C', self.repo_dir] + list(args)) + def _run_git_log(self, revision_range) -> List[GitCommit]: - log = _check_output(['git', '-C', self.repo_dir, 'log', - '--format=fuller', '--date=unix', revision_range, '--', '.']).split("\n") + log = self._run_git_cmd('log', '--format=fuller', + '--date=unix', revision_range, '--', '.').split("\n") rc: List[GitCommit] = [] cur_msg: List[str] = [] for line in log: @@ -179,6 +182,18 @@ def _run_git_log(self, revision_range) -> List[GitCommit]: def get_commit_list(self, from_ref, to_ref) -> List[GitCommit]: return self._run_git_log(f"{self.remote}/{from_ref}..{self.remote}/{to_ref}") + def get_ghstack_orig_branches(self) -> List[str]: + return [x.strip() for x in self._run_git_cmd("branch", "--remotes", "--list", self.remote + "/gh/*/orig").strip().split("\n")] + + def show_ref(self, ref) -> str: + return self._run_git_cmd("show-ref", ref).split(" ")[0] + + def merge_base(self, ref1, ref2) -> str: + return self._run_git_cmd("merge-base", ref1, ref2).strip() + + def rev_list(self, ref): + return self._run_git_cmd("rev-list", f"{self.remote}/master..{ref}").strip().split() + def build_commit_dict(commits: List[GitCommit]) -> Dict[str, GitCommit]: rc = {} @@ -358,6 +373,22 @@ def get_commits_dict(x, y): print(f'{html_url};{issue["title"]};{state}') +def analyze_stacks(repo: GitRepo) -> None: + from tqdm.contrib.concurrent import thread_map + branches = repo.get_ghstack_orig_branches() + stacks_by_author: Dict[str, List[int]] = {} + for branch,rv_commits in thread_map(lambda x: (x, repo.rev_list(x)), branches, max_workers=10): + author = branch.split("/")[2] + if author not in stacks_by_author: + stacks_by_author[author]=[] + stacks_by_author[author].append(len(rv_commits)) + for author, slen in sorted(stacks_by_author.items(), key=lambda x:len(x[1]), reverse=True): + if len(slen) == 1: + print(f"{author} has 1 stack of depth {slen[0]}") + continue + print(f"{author} has {len(slen)} stacks max depth is {max(slen)} avg depth is {sum(slen)/len(slen):.2f} mean is {slen[len(slen)//2]}") + + def parse_arguments(): from argparse import ArgumentParser parser = ArgumentParser(description="Print GitHub repo stats") @@ -375,6 +406,7 @@ def parse_arguments(): parser.add_argument("--print-reverts", action="store_true") parser.add_argument("--contributor-stats", action="store_true") parser.add_argument("--missing-in-branch", action="store_true") + parser.add_argument("--analyze-stacks", action="store_true") return parser.parse_args() @@ -392,6 +424,10 @@ def main(): repo = GitRepo(args.repo_path, remote) + if args.analyze_stacks: + analyze_stacks(repo) + return + if args.missing_in_branch: # Use milestone idx or search it along milestone titles try: diff --git a/analytics/validate_binaries.py b/analytics/validate_binaries.py index c3fd4ff2bf..65965c59ad 100644 --- a/analytics/validate_binaries.py +++ b/analytics/validate_binaries.py @@ -6,10 +6,10 @@ PLATFORMS = ["osx-64", "linux-64", "win-64"] PYTHON_VERSIONS = ["3.10", "3.9", "3.8", "3.7"] CUDA_CUDNN_VERSION = [ - ("11.5", "8.3.2"), ("11.3", "8.2.0"), ("11.1", "8.0.5"), ("10.2", "7.6.5"), ("cpu", None) + ("11.7", "8.5.0"), ("cpu", None) ] CHANNEL = "pytorch-test" -VERSION = "1.11.*" +VERSION = "1.13.*" def generate_expected_builds(platform: str) -> set: @@ -22,9 +22,6 @@ def generate_expected_builds(platform: str) -> set: for cuda_version, cudnn_version in CUDA_CUDNN_VERSION: if platform == "win-64": - if cuda_version == "10.2": - # win does not support cuda 10.2 - continue cudnn_version = "8" if cuda_version == "cpu": diff --git a/check_binary.sh b/check_binary.sh index 153fca7451..80dd1e5cac 100755 --- a/check_binary.sh +++ b/check_binary.sh @@ -22,6 +22,19 @@ set -eux -o pipefail # libtorch package. +if [[ -z ${DESIRED_PYTHON:-} ]]; then + export DESIRED_PYTHON=${MATRIX_PYTHON_VERSION:-} +fi +if [[ -z ${DESIRED_CUDA:-} ]]; then + export DESIRED_CUDA=${MATRIX_DESIRED_CUDA:-} +fi +if [[ -z ${DESIRED_DEVTOOLSET:-} ]]; then + export DESIRED_DEVTOOLSET=${MATRIX_DESIRED_DEVTOOLSET:-} +fi +if [[ -z ${PACKAGE_TYPE:-} ]]; then + export PACKAGE_TYPE=${MATRIX_PACKAGE_TYPE:-} +fi + # The install root depends on both the package type and the os # All MacOS packages use conda, even for the wheel packages. if [[ "$PACKAGE_TYPE" == libtorch ]]; then @@ -38,7 +51,7 @@ else install_root="$(dirname $(which python))/../lib/python${py_dot}/site-packages/torch/" fi -if [[ "$DESIRED_CUDA" != 'cpu' && "$DESIRED_CUDA" != *"rocm"* ]]; then +if [[ "$DESIRED_CUDA" != 'cpu' && "$DESIRED_CUDA" != 'cpu-cxx11-abi' && "$DESIRED_CUDA" != *"rocm"* ]]; then # cu90, cu92, cu100, cu101 if [[ ${#DESIRED_CUDA} -eq 4 ]]; then CUDA_VERSION="${DESIRED_CUDA:2:1}.${DESIRED_CUDA:3:1}" @@ -328,7 +341,7 @@ fi if [[ "$PACKAGE_TYPE" == 'libtorch' ]]; then echo "Checking that MKL is available" build_and_run_example_cpp check-torch-mkl -else +elif [[ "$(uname -m)" != "arm64" ]]; then if [[ "$(uname)" != 'Darwin' || "$PACKAGE_TYPE" != *wheel ]]; then echo "Checking that MKL is available" pushd /tmp @@ -366,7 +379,7 @@ if [[ "$OSTYPE" == "msys" ]]; then fi # Test that CUDA builds are setup correctly -if [[ "$DESIRED_CUDA" != 'cpu' && "$DESIRED_CUDA" != *"rocm"* ]]; then +if [[ "$DESIRED_CUDA" != 'cpu' && "$DESIRED_CUDA" != 'cpu-cxx11-abi' && "$DESIRED_CUDA" != *"rocm"* ]]; then if [[ "$PACKAGE_TYPE" == 'libtorch' ]]; then build_and_run_example_cpp check-torch-cuda else @@ -392,6 +405,9 @@ if [[ "$DESIRED_CUDA" != 'cpu' && "$DESIRED_CUDA" != *"rocm"* ]]; then echo "Checking that basic CNN works" python ${TEST_CODE_DIR}/cnn_smoke.py + echo "Test that linalg works" + python -c "import torch;x=torch.rand(3,3,device='cuda');print(torch.linalg.svd(torch.mm(x.t(), x)))" + popd fi # if libtorch fi # if cuda @@ -418,8 +434,8 @@ fi ############################################################################### # Check for C++ ABI compatibility between gcc7 and gcc9 compiled binaries ############################################################################### -if [[ "$(uname)" == 'Linux' && ("$PACKAGE_TYPE" == 'conda' || "$PACKAGE_TYPE" == 'manywheel') ]]; then +if [[ "$(uname)" == 'Linux' && ("$PACKAGE_TYPE" == 'conda' || "$PACKAGE_TYPE" == 'manywheel')]]; then pushd /tmp - python -c "import torch; exit(0 if torch._C._PYBIND11_BUILD_ABI == '_cxxabi1011' else 1)" + python -c "import torch; exit(0 if torch.compiled_with_cxx11_abi() else (0 if torch._C._PYBIND11_BUILD_ABI == '_cxxabi1011' else 1))" popd fi diff --git a/common/install_conda.sh b/common/install_conda.sh index 43dd193972..bd06075257 100644 --- a/common/install_conda.sh +++ b/common/install_conda.sh @@ -5,8 +5,11 @@ set -ex # Anaconda wget -q https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh chmod +x Miniconda3-latest-Linux-x86_64.sh -bash Miniconda3-latest-Linux-x86_64.sh -b -p /opt/conda +# NB: Manually invoke bash per https://github.com/conda/conda/issues/10431 +bash ./Miniconda3-latest-Linux-x86_64.sh -b -p /opt/conda rm Miniconda3-latest-Linux-x86_64.sh export PATH=/opt/conda/bin:$PATH -conda install -y conda-build anaconda-client git ninja +# The cmake version here needs to match with the minimum version of cmake +# supported by PyTorch (3.18). There is only 3.18.2 on anaconda +conda install -y conda-build anaconda-client git ninja cmake=3.18.2 conda remove -y --force patchelf diff --git a/common/install_cpython.sh b/common/install_cpython.sh index f393de2025..b06fe27c16 100755 --- a/common/install_cpython.sh +++ b/common/install_cpython.sh @@ -14,37 +14,35 @@ function check_var { fi } -function lex_pyver { - # Echoes Python version string padded with zeros - # Thus: - # 3.2.1 -> 003002001 - # 3 -> 003000000 - echo $1 | awk -F "." '{printf "%03d%03d%03d", $1, $2, $3}' -} - function do_cpython_build { local py_ver=$1 check_var $py_ver - local ucs_setting=$2 - check_var $ucs_setting tar -xzf Python-$py_ver.tgz pushd Python-$py_ver - if [ "$ucs_setting" = "none" ]; then - unicode_flags="" - dir_suffix="" + + local prefix="/opt/_internal/cpython-${py_ver}" + mkdir -p ${prefix}/lib + if [[ -n $(which patchelf) ]]; then + local shared_flags="--enable-shared" else - local unicode_flags="--enable-unicode=$ucs_setting" - local dir_suffix="-$ucs_setting" + local shared_flags="--disable-shared" + fi + if [[ -z "${WITH_OPENSSL+x}" ]]; then + local openssl_flags="" + else + local openssl_flags="--with-openssl=${WITH_OPENSSL} --with-openssl-rpath=auto" fi - local prefix="/opt/_internal/cpython-${py_ver}${dir_suffix}" - mkdir -p ${prefix}/lib # -Wformat added for https://bugs.python.org/issue17547 on Python 2.6 - CFLAGS="-Wformat" ./configure --prefix=${prefix} --disable-shared $unicode_flags > /dev/null + CFLAGS="-Wformat" ./configure --prefix=${prefix} ${openssl_flags} ${shared_flags} > /dev/null make -j40 > /dev/null make install > /dev/null + if [[ "${shared_flags}" == "--enable-shared" ]]; then + patchelf --set-rpath '$ORIGIN/../lib' ${prefix}/bin/python3 + fi + popd rm -rf Python-$py_ver # Some python's install as bin/python3. Make them available as @@ -61,27 +59,16 @@ function do_cpython_build { ln -s ${prefix} /opt/python/${abi_tag} } - function build_cpython { local py_ver=$1 check_var $py_ver check_var $PYTHON_DOWNLOAD_URL local py_ver_folder=$py_ver - # Only beta version of 3.11 is available right now - if [ "$py_ver" = "3.11.0" ]; then - py_ver=$py_ver"b1" - fi wget -q $PYTHON_DOWNLOAD_URL/$py_ver_folder/Python-$py_ver.tgz - if [ $(lex_pyver $py_ver) -lt $(lex_pyver 3.3) ]; then - do_cpython_build $py_ver ucs2 - do_cpython_build $py_ver ucs4 - else - do_cpython_build $py_ver none - fi + do_cpython_build $py_ver none rm -f Python-$py_ver.tgz } - function build_cpythons { check_var $GET_PIP_URL curl -sLO $GET_PIP_URL @@ -91,7 +78,6 @@ function build_cpythons { rm -f get-pip.py } - mkdir -p /opt/python mkdir -p /opt/_internal build_cpythons $CPYTHON_VERSIONS diff --git a/common/install_cuda.sh b/common/install_cuda.sh index 77d1900113..359df5b3bb 100644 --- a/common/install_cuda.sh +++ b/common/install_cuda.sh @@ -2,80 +2,6 @@ set -ex -function install_102 { - echo "Installing CUDA 10.2 and CuDNN" - rm -rf /usr/local/cuda-10.2 /usr/local/cuda - # # install CUDA 10.2 in the same container - wget -q http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_linux.run - chmod +x cuda_10.2.89_440.33.01_linux.run - ./cuda_10.2.89_440.33.01_linux.run --extract=/tmp/cuda - rm -f cuda_10.2.89_440.33.01_linux.run - mv /tmp/cuda/cuda-toolkit /usr/local/cuda-10.2 - rm -rf /tmp/cuda - rm -f /usr/local/cuda && ln -s /usr/local/cuda-10.2 /usr/local/cuda - - # install CUDA 10.2 CuDNN - # cuDNN license: https://developer.nvidia.com/cudnn/license_agreement - mkdir tmp_cudnn && cd tmp_cudnn - wget -q http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7-dev_7.6.5.32-1+cuda10.2_amd64.deb -O cudnn-dev.deb - wget -q http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7_7.6.5.32-1+cuda10.2_amd64.deb -O cudnn.deb - ar -x cudnn-dev.deb && tar -xvf data.tar.xz - ar -x cudnn.deb && tar -xvf data.tar.xz - mkdir -p cuda/include && mkdir -p cuda/lib64 - cp -a usr/include/x86_64-linux-gnu/cudnn_v7.h cuda/include/cudnn.h - cp -a usr/lib/x86_64-linux-gnu/libcudnn* cuda/lib64 - mv cuda/lib64/libcudnn_static_v7.a cuda/lib64/libcudnn_static.a - ln -s libcudnn.so.7 cuda/lib64/libcudnn.so - chmod +x cuda/lib64/*.so - cp -a cuda/include/* /usr/local/cuda/include/ - cp -a cuda/lib64/* /usr/local/cuda/lib64/ - cd .. - rm -rf tmp_cudnn - ldconfig -} - -function install_113 { - echo "Installing CUDA 11.3 and CuDNN 8.3" - rm -rf /usr/local/cuda-11.3 /usr/local/cuda - # install CUDA 11.3.1 in the same container - wget -q https://developer.download.nvidia.com/compute/cuda/11.3.1/local_installers/cuda_11.3.1_465.19.01_linux.run - chmod +x cuda_11.3.1_465.19.01_linux.run - ./cuda_11.3.1_465.19.01_linux.run --toolkit --silent - rm -f cuda_11.3.1_465.19.01_linux.run - rm -f /usr/local/cuda && ln -s /usr/local/cuda-11.3 /usr/local/cuda - - # cuDNN license: https://developer.nvidia.com/cudnn/license_agreement - mkdir tmp_cudnn && cd tmp_cudnn - wget -q https://developer.download.nvidia.com/compute/redist/cudnn/v8.3.2/local_installers/11.5/cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive.tar.xz -O cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive.tar.xz - tar xf cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive.tar.xz - cp -a cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive/include/* /usr/local/cuda/include/ - cp -a cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive/lib/* /usr/local/cuda/lib64/ - cd .. - rm -rf tmp_cudnn - ldconfig -} - -function install_115 { - echo "Installing CUDA 11.5 and CuDNN 8.3" - rm -rf /usr/local/cuda-11.5 /usr/local/cuda - # install CUDA 11.5.0 in the same container - wget -q https://developer.download.nvidia.com/compute/cuda/11.5.0/local_installers/cuda_11.5.0_495.29.05_linux.run - chmod +x cuda_11.5.0_495.29.05_linux.run - ./cuda_11.5.0_495.29.05_linux.run --toolkit --silent - rm -f cuda_11.5.0_495.29.05_linux.run - rm -f /usr/local/cuda && ln -s /usr/local/cuda-11.5 /usr/local/cuda - - # cuDNN license: https://developer.nvidia.com/cudnn/license_agreement - mkdir tmp_cudnn && cd tmp_cudnn - wget -q https://developer.download.nvidia.com/compute/redist/cudnn/v8.3.2/local_installers/11.5/cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive.tar.xz -O cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive.tar.xz - tar xf cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive.tar.xz - cp -a cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive/include/* /usr/local/cuda/include/ - cp -a cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive/lib/* /usr/local/cuda/lib64/ - cd .. - rm -rf tmp_cudnn - ldconfig -} - function install_116 { echo "Installing CUDA 11.6 and CuDNN 8.3" rm -rf /usr/local/cuda-11.6 /usr/local/cuda @@ -98,7 +24,7 @@ function install_116 { } function install_117 { - echo "Installing CUDA 11.7 and CuDNN 8.3" + echo "Installing CUDA 11.7 and CuDNN 8.5 and NCCL 2.14" rm -rf /usr/local/cuda-11.7 /usr/local/cuda # install CUDA 11.7.0 in the same container wget -q https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda_11.7.0_515.43.04_linux.run @@ -116,79 +42,56 @@ function install_117 { cd .. rm -rf tmp_cudnn ldconfig -} - -function prune_102 { - echo "Pruning CUDA 10.2 and CuDNN" - ##################################################################################### - # CUDA 10.2 prune static libs - ##################################################################################### - export NVPRUNE="/usr/local/cuda-10.2/bin/nvprune" - export CUDA_LIB_DIR="/usr/local/cuda-10.2/lib64" - - export GENCODE="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75" - export GENCODE_CUDNN="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75" - - if [[ -n "$OVERRIDE_GENCODE" ]]; then - export GENCODE=$OVERRIDE_GENCODE - fi - - # all CUDA libs except CuDNN and CuBLAS (cudnn and cublas need arch 3.7 included) - ls $CUDA_LIB_DIR/ | grep "\.a" | grep -v "culibos" | grep -v "cudart" | grep -v "cudnn" | grep -v "cublas" | grep -v "metis" \ - | xargs -I {} bash -c \ - "echo {} && $NVPRUNE $GENCODE $CUDA_LIB_DIR/{} -o $CUDA_LIB_DIR/{}" - - # prune CuDNN and CuBLAS - $NVPRUNE $GENCODE_CUDNN $CUDA_LIB_DIR/libcudnn_static.a -o $CUDA_LIB_DIR/libcudnn_static.a - $NVPRUNE $GENCODE_CUDNN $CUDA_LIB_DIR/libcublas_static.a -o $CUDA_LIB_DIR/libcublas_static.a - $NVPRUNE $GENCODE_CUDNN $CUDA_LIB_DIR/libcublasLt_static.a -o $CUDA_LIB_DIR/libcublasLt_static.a - - ##################################################################################### - # CUDA 10.2 prune visual tools - ##################################################################################### - export CUDA_BASE="/usr/local/cuda-10.2/" - rm -rf $CUDA_BASE/libnsight $CUDA_BASE/libnvvp $CUDA_BASE/nsightee_plugins $CUDA_BASE/nsight-compute-2019.5.0 $CUDA_BASE/nsight-systems-2019.5.2 + # NCCL license: https://docs.nvidia.com/deeplearning/nccl/#licenses + mkdir tmp_nccl && cd tmp_nccl + wget -q https://developer.download.nvidia.com/compute/redist/nccl/v2.14/nccl_2.14.3-1+cuda11.7_x86_64.txz + tar xf nccl_2.14.3-1+cuda11.7_x86_64.txz + cp -a nccl_2.14.3-1+cuda11.7_x86_64/include/* /usr/local/cuda/include/ + cp -a nccl_2.14.3-1+cuda11.7_x86_64/lib/* /usr/local/cuda/lib64/ + cd .. + rm -rf tmp_nccl + ldconfig } -function prune_113 { - echo "Pruning CUDA 11.3 and CuDNN" - ##################################################################################### - # CUDA 11.3 prune static libs - ##################################################################################### - export NVPRUNE="/usr/local/cuda-11.3/bin/nvprune" - export CUDA_LIB_DIR="/usr/local/cuda-11.3/lib64" - - export GENCODE="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86" - export GENCODE_CUDNN="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86" - - if [[ -n "$OVERRIDE_GENCODE" ]]; then - export GENCODE=$OVERRIDE_GENCODE - fi - - # all CUDA libs except CuDNN and CuBLAS (cudnn and cublas need arch 3.7 included) - ls $CUDA_LIB_DIR/ | grep "\.a" | grep -v "culibos" | grep -v "cudart" | grep -v "cudnn" | grep -v "cublas" | grep -v "metis" \ - | xargs -I {} bash -c \ - "echo {} && $NVPRUNE $GENCODE $CUDA_LIB_DIR/{} -o $CUDA_LIB_DIR/{}" +function install_118 { + echo "Installing CUDA 11.8 and cuDNN 8.7 and NCCL 2.15" + rm -rf /usr/local/cuda-11.8 /usr/local/cuda + # install CUDA 11.8.0 in the same container + wget -q https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run + chmod +x cuda_11.8.0_520.61.05_linux.run + ./cuda_11.8.0_520.61.05_linux.run --toolkit --silent + rm -f cuda_11.8.0_520.61.05_linux.run + rm -f /usr/local/cuda && ln -s /usr/local/cuda-11.8 /usr/local/cuda - # prune CuDNN and CuBLAS - $NVPRUNE $GENCODE_CUDNN $CUDA_LIB_DIR/libcublas_static.a -o $CUDA_LIB_DIR/libcublas_static.a - $NVPRUNE $GENCODE_CUDNN $CUDA_LIB_DIR/libcublasLt_static.a -o $CUDA_LIB_DIR/libcublasLt_static.a + # cuDNN license: https://developer.nvidia.com/cudnn/license_agreement + mkdir tmp_cudnn && cd tmp_cudnn + wget -q https://developer.download.nvidia.com/compute/redist/cudnn/v8.7.0/local_installers/11.8/cudnn-linux-x86_64-8.7.0.84_cuda11-archive.tar.xz -O cudnn-linux-x86_64-8.7.0.84_cuda11-archive.tar.xz + tar xf cudnn-linux-x86_64-8.7.0.84_cuda11-archive.tar.xz + cp -a cudnn-linux-x86_64-8.7.0.84_cuda11-archive/include/* /usr/local/cuda/include/ + cp -a cudnn-linux-x86_64-8.7.0.84_cuda11-archive/lib/* /usr/local/cuda/lib64/ + cd .. + rm -rf tmp_cudnn + ldconfig - ##################################################################################### - # CUDA 11.3 prune visual tools - ##################################################################################### - export CUDA_BASE="/usr/local/cuda-11.3/" - rm -rf $CUDA_BASE/libnvvp $CUDA_BASE/nsightee_plugins $CUDA_BASE/nsight-compute-2021.1.0 $CUDA_BASE/nsight-systems-2021.1.3 + # NCCL license: https://docs.nvidia.com/deeplearning/nccl/#licenses + mkdir tmp_nccl && cd tmp_nccl + wget -q https://developer.download.nvidia.com/compute/redist/nccl/v2.15.5/nccl_2.15.5-1+cuda11.8_x86_64.txz + tar xf nccl_2.15.5-1+cuda11.8_x86_64.txz + cp -a nccl_2.15.5-1+cuda11.8_x86_64/include/* /usr/local/cuda/include/ + cp -a nccl_2.15.5-1+cuda11.8_x86_64/lib/* /usr/local/cuda/lib64/ + cd .. + rm -rf tmp_nccl + ldconfig } -function prune_115 { - echo "Pruning CUDA 11.5 and CuDNN" +function prune_116 { + echo "Pruning CUDA 11.6 and CuDNN" ##################################################################################### - # CUDA 11.3 prune static libs + # CUDA 11.6 prune static libs ##################################################################################### - export NVPRUNE="/usr/local/cuda-11.5/bin/nvprune" - export CUDA_LIB_DIR="/usr/local/cuda-11.5/lib64" + export NVPRUNE="/usr/local/cuda-11.6/bin/nvprune" + export CUDA_LIB_DIR="/usr/local/cuda-11.6/lib64" export GENCODE="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86" export GENCODE_CUDNN="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86" @@ -200,26 +103,26 @@ function prune_115 { # all CUDA libs except CuDNN and CuBLAS (cudnn and cublas need arch 3.7 included) ls $CUDA_LIB_DIR/ | grep "\.a" | grep -v "culibos" | grep -v "cudart" | grep -v "cudnn" | grep -v "cublas" | grep -v "metis" \ | xargs -I {} bash -c \ - "echo {} && $NVPRUNE $GENCODE $CUDA_LIB_DIR/{} -o $CUDA_LIB_DIR/{}" + "echo {} && $NVPRUNE $GENCODE $CUDA_LIB_DIR/{} -o $CUDA_LIB_DIR/{}" # prune CuDNN and CuBLAS $NVPRUNE $GENCODE_CUDNN $CUDA_LIB_DIR/libcublas_static.a -o $CUDA_LIB_DIR/libcublas_static.a $NVPRUNE $GENCODE_CUDNN $CUDA_LIB_DIR/libcublasLt_static.a -o $CUDA_LIB_DIR/libcublasLt_static.a ##################################################################################### - # CUDA 11.5 prune visual tools + # CUDA 11.6 prune visual tools ##################################################################################### - export CUDA_BASE="/usr/local/cuda-11.5/" - rm -rf $CUDA_BASE/libnvvp $CUDA_BASE/nsightee_plugins $CUDA_BASE/nsight-compute-2021.3.0 $CUDA_BASE/nsight-systems-2021.3.3 + export CUDA_BASE="/usr/local/cuda-11.6/" + rm -rf $CUDA_BASE/libnvvp $CUDA_BASE/nsightee_plugins $CUDA_BASE/nsight-compute-2022.1.1 $CUDA_BASE/nsight-systems-2021.5.2 } -function prune_116 { - echo "Pruning CUDA 11.6 and CuDNN" +function prune_117 { + echo "Pruning CUDA 11.7 and CuDNN" ##################################################################################### - # CUDA 11.6 prune static libs + # CUDA 11.7 prune static libs ##################################################################################### - export NVPRUNE="/usr/local/cuda-11.6/bin/nvprune" - export CUDA_LIB_DIR="/usr/local/cuda-11.6/lib64" + export NVPRUNE="/usr/local/cuda-11.7/bin/nvprune" + export CUDA_LIB_DIR="/usr/local/cuda-11.7/lib64" export GENCODE="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86" export GENCODE_CUDNN="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86" @@ -240,20 +143,20 @@ function prune_116 { ##################################################################################### # CUDA 11.6 prune visual tools ##################################################################################### - export CUDA_BASE="/usr/local/cuda-11.6/" - rm -rf $CUDA_BASE/libnvvp $CUDA_BASE/nsightee_plugins $CUDA_BASE/nsight-compute-2022.1.1 $CUDA_BASE/nsight-systems-2021.5.2 + export CUDA_BASE="/usr/local/cuda-11.7/" + rm -rf $CUDA_BASE/libnvvp $CUDA_BASE/nsightee_plugins $CUDA_BASE/nsight-compute-2022.2.0 $CUDA_BASE/nsight-systems-2022.1.3 } -function prune_117 { - echo "Pruning CUDA 11.7 and CuDNN" +function prune_118 { + echo "Pruning CUDA 11.8 and cuDNN" ##################################################################################### - # CUDA 11.7 prune static libs + # CUDA 11.8 prune static libs ##################################################################################### - export NVPRUNE="/usr/local/cuda-11.7/bin/nvprune" - export CUDA_LIB_DIR="/usr/local/cuda-11.7/lib64" + export NVPRUNE="/usr/local/cuda-11.8/bin/nvprune" + export CUDA_LIB_DIR="/usr/local/cuda-11.8/lib64" - export GENCODE="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86" - export GENCODE_CUDNN="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86" + export GENCODE="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_90,code=sm_90" + export GENCODE_CUDNN="-gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_90,code=sm_90" if [[ -n "$OVERRIDE_GENCODE" ]]; then export GENCODE=$OVERRIDE_GENCODE @@ -269,26 +172,22 @@ function prune_117 { $NVPRUNE $GENCODE_CUDNN $CUDA_LIB_DIR/libcublasLt_static.a -o $CUDA_LIB_DIR/libcublasLt_static.a ##################################################################################### - # CUDA 11.6 prune visual tools + # CUDA 11.8 prune visual tools ##################################################################################### - export CUDA_BASE="/usr/local/cuda-11.7/" - rm -rf $CUDA_BASE/libnvvp $CUDA_BASE/nsightee_plugins $CUDA_BASE/nsight-compute-2022.2.0 $CUDA_BASE/nsight-systems-2022.1.3 + export CUDA_BASE="/usr/local/cuda-11.8/" + rm -rf $CUDA_BASE/libnvvp $CUDA_BASE/nsightee_plugins $CUDA_BASE/nsight-compute-2022.3.0 $CUDA_BASE/nsight-systems-2022.4.2/ } # idiomatic parameter and option handling in sh while test $# -gt 0 do case "$1" in - 10.2) install_102; prune_102 - ;; - 11.3) install_113; prune_113 - ;; - 11.5) install_115; prune_115 - ;; 11.6) install_116; prune_116 ;; 11.7) install_117; prune_117 ;; + 11.8) install_118; prune_118 + ;; *) echo "bad argument $1"; exit 1 ;; esac diff --git a/common/install_magma.sh b/common/install_magma.sh index 5d14dbfe2f..b524c920e9 100644 --- a/common/install_magma.sh +++ b/common/install_magma.sh @@ -7,17 +7,10 @@ MAGMA_VERSION="2.5.2" function do_install() { cuda_version=$1 cuda_version_nodot=${1/./} - - if [[ ${cuda_version_nodot} == 116 ]]; then - MAGMA_VERSION="2.6.1" - magma_archive="magma-cuda${cuda_version_nodot}-${MAGMA_VERSION}-0.tar.bz2" - elif [[ ${cuda_version_nodot} == 117 ]]; then - MAGMA_VERSION="2.6.1" - magma_archive="magma-cuda${cuda_version_nodot}-${MAGMA_VERSION}-0.tar.bz2" - else - magma_archive="magma-cuda${cuda_version_nodot}-${MAGMA_VERSION}-1.tar.bz2" - fi - + + MAGMA_VERSION="2.6.1" + magma_archive="magma-cuda${cuda_version_nodot}-${MAGMA_VERSION}-1.tar.bz2" + cuda_dir="/usr/local/cuda-${cuda_version}" ( set -x diff --git a/common/install_miopen.sh b/common/install_miopen.sh index 27521c429d..a5166c0974 100644 --- a/common/install_miopen.sh +++ b/common/install_miopen.sh @@ -33,8 +33,9 @@ if [[ $ROCM_INT -lt 40001 ]]; then exit 0 fi +# CHANGED: Do not uninstall. To avoid out of disk space issues, we will copy lib over existing. # Uninstall existing package, to avoid errors during later yum install indicating packages did not change. -yum remove -y miopen-hip +#yum remove -y miopen-hip # Function to retry functions that sometimes timeout or have flaky failures retry () { @@ -91,8 +92,25 @@ fi git clone https://github.com/ROCmSoftwarePlatform/MIOpen -b ${MIOPEN_BRANCH} pushd MIOpen +# remove .git to save disk space ince CI runner was running out +rm -rf .git +# Don't build MLIR to save docker build time +# since we are disabling MLIR backend for MIOpen anyway +if [[ $ROCM_INT -ge 50400 ]] && [[ $ROCM_INT -lt 50500 ]]; then + sed -i '/rocMLIR/d' requirements.txt +elif [[ $ROCM_INT -ge 50200 ]] && [[ $ROCM_INT -lt 50400 ]]; then + sed -i '/llvm-project-mlir/d' requirements.txt +fi ## MIOpen minimum requirements cmake -P install_deps.cmake --minimum + +# clean up since CI runner was running out of disk space +rm -rf /tmp/* +yum clean all +rm -rf /var/cache/yum +rm -rf /var/lib/yum/yumdb +rm -rf /var/lib/yum/history + ## Build MIOpen mkdir -p build cd build @@ -101,13 +119,19 @@ PKG_CONFIG_PATH=/usr/local/lib/pkgconfig CXX=${ROCM_INSTALL_PATH}/llvm/bin/clang ${MIOPEN_CMAKE_DB_FLAGS} \ -DCMAKE_PREFIX_PATH="${ROCM_INSTALL_PATH}/hip;${ROCM_INSTALL_PATH}" make MIOpen -j $(nproc) -make -j $(nproc) package -yum install -y miopen-*.rpm + +# CHANGED: Do not build package. +# Build MIOpen package +#make -j $(nproc) package + +# clean up since CI runner was running out of disk space +rm -rf /usr/local/cget + +# CHANGED: Do not install package, just copy lib over existing. +#yum install -y miopen-*.rpm +dest=$(ls ${ROCM_INSTALL_PATH}/lib/libMIOpen.so.1.0.*) +rm -f ${dest} +cp lib/libMIOpen.so.1.0 ${dest} + popd rm -rf MIOpen - -# Cleanup -yum clean all -rm -rf /var/cache/yum -rm -rf /var/lib/yum/yumdb -rm -rf /var/lib/yum/history diff --git a/common/install_mkl.sh b/common/install_mkl.sh index 5ebdd94b1b..5889dc1f0e 100644 --- a/common/install_mkl.sh +++ b/common/install_mkl.sh @@ -3,8 +3,8 @@ set -ex # MKL -MKL_VERSION=2020.0 -MKL_BUILD=166 +MKL_VERSION=2022.2.1 +MKL_BUILD=16993 mkdir -p /opt/intel/lib pushd /tmp curl -fsSL https://anaconda.org/intel/mkl-static/${MKL_VERSION}/download/linux-64/mkl-static-${MKL_VERSION}-intel_${MKL_BUILD}.tar.bz2 | tar xjv diff --git a/common/install_patchelf.sh b/common/install_patchelf.sh index 032e3cc27a..37b69415e8 100644 --- a/common/install_patchelf.sh +++ b/common/install_patchelf.sh @@ -2,7 +2,9 @@ set -ex -git clone https://github.com/NixOS/patchelf +# Pin the version to latest release 0.17.2, building newer commit starts +# to fail on the current image +git clone -b 0.17.2 --single-branch https://github.com/NixOS/patchelf cd patchelf sed -i 's/serial/parallel/g' configure.ac ./bootstrap.sh diff --git a/common/install_rocm.sh b/common/install_rocm.sh index d4352c21c7..4323cebd29 100644 --- a/common/install_rocm.sh +++ b/common/install_rocm.sh @@ -47,6 +47,10 @@ install_ubuntu() { ROCM_REPO="xenial" fi + if [[ $(ver $ROCM_VERSION) -ge $(ver 5.3) ]]; then + ROCM_REPO="${UBUNTU_VERSION_NAME}" + fi + # Add rocm repository wget -qO - http://repo.radeon.com/rocm/rocm.gpg.key | apt-key add - local rocm_baseurl="http://repo.radeon.com/rocm/apt/${ROCM_VERSION}" diff --git a/common/install_rocm_magma.sh b/common/install_rocm_magma.sh index c651a6e4e2..00540fbecd 100644 --- a/common/install_rocm_magma.sh +++ b/common/install_rocm_magma.sh @@ -37,5 +37,8 @@ make -f make.gen.hipMAGMA -j $(nproc) LANG=C.UTF-8 make lib/libmagma.so -j $(nproc) MKLROOT="${MKLROOT}" make testing/testing_dgemm -j $(nproc) MKLROOT="${MKLROOT}" popd -mv magma /opt/rocm +mkdir -p /opt/rocm/magma +mv magma/include /opt/rocm/magma +mv magma/lib /opt/rocm/magma +rm -rf magma diff --git a/conda/Dockerfile b/conda/Dockerfile index f4f4c834a6..c65e1ad99e 100644 --- a/conda/Dockerfile +++ b/conda/Dockerfile @@ -41,20 +41,12 @@ RUN bash ./install_conda.sh && rm install_conda.sh # Install CUDA FROM base as cuda +ARG CUDA_VERSION=10.2 RUN rm -rf /usr/local/cuda-* ADD ./common/install_cuda.sh install_cuda.sh - -FROM cuda as cuda10.2 -RUN bash ./install_cuda.sh 10.2 -ENV DESIRED_CUDA=10.2 - -FROM cuda as cuda11.3 -RUN bash ./install_cuda.sh 11.3 -ENV DESIRED_CUDA=11.3 - -FROM cuda as cuda11.5 -RUN bash ./install_cuda.sh 11.5 -ENV DESIRED_CUDA=11.5 +ENV CUDA_HOME=/usr/local/cuda-${CUDA_VERSION} +# Make things in our path by default +ENV PATH=/usr/local/cuda-${CUDA_VERSION}/bin:$PATH FROM cuda as cuda11.6 RUN bash ./install_cuda.sh 11.6 @@ -64,17 +56,19 @@ FROM cuda as cuda11.7 RUN bash ./install_cuda.sh 11.7 ENV DESIRED_CUDA=11.7 +FROM cuda as cuda11.8 +RUN bash ./install_cuda.sh 11.8 +ENV DESIRED_CUDA=11.8 + # Install MNIST test data FROM base as mnist ADD ./common/install_mnist.sh install_mnist.sh RUN bash ./install_mnist.sh FROM base as all_cuda -COPY --from=cuda10.2 /usr/local/cuda-10.2 /usr/local/cuda-10.2 -COPY --from=cuda11.3 /usr/local/cuda-11.3 /usr/local/cuda-11.3 -COPY --from=cuda11.5 /usr/local/cuda-11.5 /usr/local/cuda-11.5 COPY --from=cuda11.6 /usr/local/cuda-11.6 /usr/local/cuda-11.6 COPY --from=cuda11.7 /usr/local/cuda-11.7 /usr/local/cuda-11.7 +COPY --from=cuda11.8 /usr/local/cuda-11.8 /usr/local/cuda-11.8 FROM ${BASE_TARGET} as final # Install LLVM diff --git a/conda/build_all_docker.sh b/conda/build_all_docker.sh index bc43976750..1dc5ffe4f9 100755 --- a/conda/build_all_docker.sh +++ b/conda/build_all_docker.sh @@ -4,6 +4,6 @@ set -eou pipefail TOPDIR=$(git rev-parse --show-toplevel) -for CUDA_VERSION in 11.7 11.6 11.5 11.3 10.2 cpu; do +for CUDA_VERSION in 11.8 11.7 11.6 cpu; do CUDA_VERSION="${CUDA_VERSION}" conda/build_docker.sh done diff --git a/conda/build_docker.sh b/conda/build_docker.sh index db7c5be62c..a3385ff38c 100755 --- a/conda/build_docker.sh +++ b/conda/build_docker.sh @@ -52,6 +52,11 @@ if [[ "${DOCKER_TAG}" =~ ^cuda* ]]; then set -x docker tag ${DOCKER_IMAGE} "pytorch/conda-builder:cuda${CUDA_VERSION/./}" ) + # Test that we're using the right CUDA compiler + ( + set -x + docker run --rm "${DOCKER_IMAGE}" nvcc --version | grep "cuda_${CUDA_VERSION}" + ) fi if [[ -n ${GITHUB_REF} ]]; then diff --git a/conda/build_pytorch.sh b/conda/build_pytorch.sh index e430538c3f..30986b4088 100755 --- a/conda/build_pytorch.sh +++ b/conda/build_pytorch.sh @@ -31,7 +31,7 @@ retry () { $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) } -# Parse arguments and determmine version +# Parse arguments and determine version ########################################################### if [[ -n "$DESIRED_CUDA" && -n "$PYTORCH_BUILD_VERSION" && -n "$PYTORCH_BUILD_NUMBER" ]]; then desired_cuda="$DESIRED_CUDA" @@ -106,7 +106,7 @@ if [[ -z "$DESIRED_PYTHON" ]]; then fi fi if [[ "$OSTYPE" == "darwin"* ]]; then - DEVELOPER_DIR=/Applications/Xcode9.app/Contents/Developer + DEVELOPER_DIR=/Applications/Xcode_13.3.1.app/Contents/Developer fi if [[ "$desired_cuda" == 'cpu' ]]; then cpu_only=1 @@ -190,7 +190,7 @@ if [[ ! -d "$pytorch_rootdir" ]]; then popd fi pushd "$pytorch_rootdir" -git submodule update --init --recursive --jobs 0 +git submodule update --init --recursive echo "Using Pytorch from " git --no-pager log --max-count 1 popd @@ -207,8 +207,6 @@ if [[ "$(uname)" == 'Darwin' ]]; then rm "$miniconda_sh" export PATH="$tmp_conda/bin:$PATH" retry conda install -yq conda-build - # Install py-lief=0.12.0 containing https://github.com/lief-project/LIEF/pull/579 to speed up the builds - retry conda install -yq py-lief==0.12.0 -c malfet elif [[ "$OSTYPE" == "msys" ]]; then export tmp_conda="${WIN_PACKAGE_WORK_DIR}\\conda" export miniconda_exe="${WIN_PACKAGE_WORK_DIR}\\miniconda.exe" @@ -245,12 +243,13 @@ fi meta_yaml="$build_folder/meta.yaml" echo "Using conda-build folder $build_folder" -# Switch between CPU or CUDA configerations +# Switch between CPU or CUDA configurations ########################################################### build_string_suffix="$PYTORCH_BUILD_NUMBER" if [[ -n "$cpu_only" ]]; then export USE_CUDA=0 export CONDA_CUDATOOLKIT_CONSTRAINT="" + export CONDA_TRITON_CONSTRAINT="" export MAGMA_PACKAGE="" export CUDA_VERSION="0.0" export CUDNN_VERSION="0.0" @@ -266,22 +265,24 @@ else . ./switch_cuda_version.sh "$desired_cuda" # TODO, simplify after anaconda fixes their cudatoolkit versioning inconsistency. # see: https://github.com/conda-forge/conda-forge.github.io/issues/687#issuecomment-460086164 - if [[ "$desired_cuda" == "11.7" ]]; then - export CONDA_CUDATOOLKIT_CONSTRAINT=" - pytorch-cuda >=11.7,<11.8 # [not osx]" - export MAGMA_PACKAGE=" - magma-cuda117 # [not osx and not win]" + if [[ "$desired_cuda" == "11.8" ]]; then + export CONDA_CUDATOOLKIT_CONSTRAINT=" - pytorch-cuda >=11.8,<11.9 # [not osx]" + export MAGMA_PACKAGE=" - magma-cuda118 # [not osx and not win]" + elif [[ "$desired_cuda" == "11.7" ]]; then + export CONDA_CUDATOOLKIT_CONSTRAINT=" - pytorch-cuda >=11.7,<11.8 # [not osx]" + export MAGMA_PACKAGE=" - magma-cuda117 # [not osx and not win]" elif [[ "$desired_cuda" == "11.6" ]]; then export CONDA_CUDATOOLKIT_CONSTRAINT=" - pytorch-cuda >=11.6,<11.7 # [not osx]" export MAGMA_PACKAGE=" - magma-cuda116 # [not osx and not win]" - elif [[ "$desired_cuda" == "11.3" ]]; then - export CONDA_CUDATOOLKIT_CONSTRAINT=" - cudatoolkit >=11.3,<11.4 # [not osx]" - export MAGMA_PACKAGE=" - magma-cuda113 # [not osx and not win]" - elif [[ "$desired_cuda" == "10.2" ]]; then - export CONDA_CUDATOOLKIT_CONSTRAINT=" - cudatoolkit >=10.2,<10.3 # [not osx]" - export MAGMA_PACKAGE=" - magma-cuda102 # [not osx and not win]" else echo "unhandled desired_cuda: $desired_cuda" exit 1 fi + if [[ "$OSTYPE" != "msys" ]]; then + # TODO: Remove me when Triton has a proper release channel + TRITON_SHORTHASH=$(cut -c1-10 $pytorch_rootdir/.github/ci_commit_pins/triton.txt) + export CONDA_TRITON_CONSTRAINT=" - torchtriton==2.1.0+${TRITON_SHORTHASH}" + fi build_string_suffix="cuda${CUDA_VERSION}_cudnn${CUDNN_VERSION}_${build_string_suffix}" fi @@ -298,6 +299,12 @@ else export CONDA_BUILD_EXTRA_ARGS="" fi +if [[ "$DESIRED_PYTHON" == "3.11" ]]; then + # TODO: Remove me when numpy is available in default channel + # or copy numpy to pytorch channel + export CONDA_BUILD_EXTRA_ARGS="-c malfet ${CONDA_BUILD_EXTRA_ARGS}" +fi + # Build PyTorch with Gloo's TCP_TLS transport if [[ "$(uname)" == 'Linux' ]]; then export USE_GLOO_WITH_OPENSSL=1 @@ -339,13 +346,14 @@ for py_ver in "${DESIRED_PYTHON[@]}"; do # Build the package echo "Build $build_folder for Python version $py_ver" conda config --set anaconda_upload no - conda install -y conda-package-handling - # NS: To be removed after conda docker images are updated - conda update -y conda-build + conda install -y conda-package-handling conda==22.9.0 if [[ "$OSTYPE" == "msys" ]]; then # Don't run tests on windows (they were ignored mostly anyways) NO_TEST="--no-test" + else + # NS: To be removed after conda docker images are updated + conda update -y conda-build fi echo "Calling conda-build at $(date)" @@ -386,7 +394,18 @@ for py_ver in "${DESIRED_PYTHON[@]}"; do # Install the built package and run tests, unless it's for mac cross compiled arm64 if [[ -z "$CROSS_COMPILE_ARM64" ]]; then - conda install -y "$built_package" + # Install the package as if from local repo instead of tar.bz2 directly in order + # to trigger runtime dependency installation. See https://github.com/conda/conda/issues/1884 + # Notes: + # - pytorch-nightly is included to install torchtriton + # - nvidia is included for cuda builds, there's no harm in listing the channel for cpu builds + if [[ "$OSTYPE" == "msys" ]]; then + # note the extra slash: `pwd -W` returns `c:/path/to/dir`, we need to add an extra slash for the URI + local_channel="/$(pwd -W)/$output_folder" + else + local_channel="$(pwd)/$output_folder" + fi + conda install -y -c "file://$local_channel" pytorch==$PYTORCH_BUILD_VERSION -c pytorch -c numba/label/dev -c pytorch-nightly -c nvidia echo "$(date) :: Running tests" pushd "$pytorch_rootdir" diff --git a/conda/build_vision.sh b/conda/build_vision.sh deleted file mode 100755 index 3061e4740b..0000000000 --- a/conda/build_vision.sh +++ /dev/null @@ -1,94 +0,0 @@ -#!/usr/bin/env bash -if [[ -x "/remote/anaconda_token" ]]; then - . /remote/anaconda_token || true -fi - -set -ex - -# Function to retry functions that sometimes timeout or have flaky failures -retry () { - $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) -} - -if [ "$#" -ne 1 ]; then - echo "Illegal number of parameters. Pass cuda version" - echo "CUDA version should be M.m with no dot, e.g. '8.0' or 'cpu'" - exit 1 -fi -desired_cuda="$1" - -export TORCHVISION_BUILD_VERSION="0.3.0" -export TORCHVISION_BUILD_NUMBER=1 - -SOURCE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" - -if [[ -z "$WIN_PACKAGE_WORK_DIR" ]]; then - WIN_PACKAGE_WORK_DIR="$(echo $(pwd -W) | tr '/' '\\')\\tmp_conda_$(date +%H%M%S)" -fi - -if [[ "$OSTYPE" == "msys" ]]; then - mkdir -p "$WIN_PACKAGE_WORK_DIR" || true - vision_rootdir="$(realpath ${WIN_PACKAGE_WORK_DIR})/torchvision-src" - git config --system core.longpaths true -else - vision_rootdir="$(pwd)/torchvision-src" -fi - -if [[ ! -d "$vision_rootdir" ]]; then - rm -rf "$vision_rootdir" - git clone "https://github.com/pytorch/vision" "$vision_rootdir" - pushd "$vision_rootdir" - git checkout v$TORCHVISION_BUILD_VERSION - popd -fi - -cd "$SOURCE_DIR" - -if [[ "$OSTYPE" == "msys" ]]; then - export tmp_conda="${WIN_PACKAGE_WORK_DIR}\\conda" - export miniconda_exe="${WIN_PACKAGE_WORK_DIR}\\miniconda.exe" - rm -rf "$tmp_conda" - rm -f "$miniconda_exe" - curl -sSk https://repo.continuum.io/miniconda/Miniconda3-latest-Windows-x86_64.exe -o "$miniconda_exe" - "$SOURCE_DIR/install_conda.bat" && rm "$miniconda_exe" - pushd $tmp_conda - export PATH="$(pwd):$(pwd)/Library/usr/bin:$(pwd)/Library/bin:$(pwd)/Scripts:$(pwd)/bin:$PATH" - popd - # We have to skip 3.17 because of the following bug. - # https://github.com/conda/conda-build/issues/3285 - retry conda install -yq conda-build -fi - -ANACONDA_USER=pytorch -conda config --set anaconda_upload no - - -export TORCHVISION_PACKAGE_SUFFIX="" -if [[ "$desired_cuda" == 'cpu' ]]; then - export CONDA_CUDATOOLKIT_CONSTRAINT="" - export CUDA_VERSION="None" - if [[ "$OSTYPE" != "darwin"* ]]; then - export TORCHVISION_PACKAGE_SUFFIX="-cpu" - fi -else - . ./switch_cuda_version.sh $desired_cuda - if [[ "$desired_cuda" == "10.0" ]]; then - export CONDA_CUDATOOLKIT_CONSTRAINT=" - cudatoolkit >=10.0,<10.1 # [not osx]" - elif [[ "$desired_cuda" == "9.0" ]]; then - export CONDA_CUDATOOLKIT_CONSTRAINT=" - cudatoolkit >=9.0,<9.1 # [not osx]" - else - echo "unhandled desired_cuda: $desired_cuda" - exit 1 - fi -fi - -if [[ "$OSTYPE" == "msys" ]]; then - time conda build -c $ANACONDA_USER --no-anaconda-upload vs2017 -else - time conda build -c $ANACONDA_USER --no-anaconda-upload --python 2.7 torchvision -fi -time conda build -c $ANACONDA_USER --no-anaconda-upload --python 3.5 torchvision -time conda build -c $ANACONDA_USER --no-anaconda-upload --python 3.6 torchvision -time conda build -c $ANACONDA_USER --no-anaconda-upload --python 3.7 torchvision - -set +e diff --git a/conda/debugging_pytorch.sh b/conda/debugging_pytorch.sh index e79567acb5..4cce4f225e 100644 --- a/conda/debugging_pytorch.sh +++ b/conda/debugging_pytorch.sh @@ -14,7 +14,7 @@ export USE_CUDA_STATIC_LINK=1 . ./switch_cuda_version.sh 9.0 -conda install -y cmake numpy=1.17 setuptools pyyaml cffi mkl=2018 mkl-include typing_extension ninja magma-cuda80 -c pytorch +conda install -y cmake numpy=1.17 setuptools pyyaml mkl=2018 mkl-include typing_extension ninja magma-cuda80 -c pytorch export CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" git clone https://github.com/pytorch/pytorch -b nightly2 --recursive diff --git a/conda/pytorch-cuda/conda_build_config.yaml b/conda/pytorch-cuda/conda_build_config.yaml index 802638b8eb..67d14f2b17 100644 --- a/conda/pytorch-cuda/conda_build_config.yaml +++ b/conda/pytorch-cuda/conda_build_config.yaml @@ -1,3 +1,7 @@ version: - 11.6 - 11.7 + - 11.8 +target_platform: + - win-64 + - linux-64 diff --git a/conda/pytorch-cuda/meta.yaml b/conda/pytorch-cuda/meta.yaml index 92c970654f..ecb438ca86 100644 --- a/conda/pytorch-cuda/meta.yaml +++ b/conda/pytorch-cuda/meta.yaml @@ -1,7 +1,40 @@ -{% set build = 0 %} +# Package to manage cuda version in PyTorch. +# +# Windows anaconda packages are packaged differently, +# All dlls are kept within *-dev packages hence we need +# include the dev packages for Windows see: +# https://github.com/pytorch/vision/issues/7185#issuecomment-1420002413 +# +# Please note: Build number should be advanced with +# every deployment. After the deployment to production +# use following links to validate the correctness of +# deployment: +# https://conda.anaconda.org/pytorch/noarch/ +# https://conda.anaconda.org/pytorch/noarch/repodata.json +{% set build = 3 %} {% set cuda_constraints=">=11.6,<11.7" %} +{% set libcufft_constraints=">=10.7.0.55,<10.7.2.50" %} +{% set libcublas_constraints=">=11.8.1.74,<11.10.1.25" %} +{% set libcusolver_constraints=">=11.3.2.55,<11.3.5.50" %} +{% set libcusparse_constraints=">=11.7.1.55,<11.7.3.50" %} +{% set libnpp_constraints=">=11.6.0.55,<11.7.3.21" %} +{% set libnvjpeg_constraints=">=11.6.0.55,<11.7.2.34" %} {% if version == '11.7' %} {% set cuda_constraints=">=11.7,<11.8" %} +{% set libcufft_constraints=">=10.7.2.50,<10.9.0.58" %} +{% set libcublas_constraints=">=11.10.1.25,<11.11.3.6" %} +{% set libcusolver_constraints=">=11.3.5.50,<11.4.1.48" %} +{% set libcusparse_constraints=">=11.7.3.50,<11.7.5.86" %} +{% set libnpp_constraints=">=11.7.3.21,<11.8.0.86" %} +{% set libnvjpeg_constraints=">=11.7.2.34,<11.9.0.86" %} +{% elif version == '11.8' %} +{% set cuda_constraints=">=11.8,<12.0" %} +{% set libcufft_constraints=">=10.9.0.58,<11.0.0.21" %} +{% set libcublas_constraints=">=11.11.3.6,<12.0.1.189" %} +{% set libcusolver_constraints=">=11.4.1.48,<11.4.2.57" %} +{% set libcusparse_constraints=">=11.7.5.86,<12.0.0.76" %} +{% set libnpp_constraints=">=11.8.0.86,<12.0.0.30" %} +{% set libnvjpeg_constraints=">=11.9.0.86,<12.0.0.28" %} {% endif %} package: @@ -9,33 +42,29 @@ package: version: {{ version }} build: number: {{ build }} - noarch: generic requirements: run: - - cuda={{ version }} - run_constrained: - - cuda-cccl {{ cuda_constraints }} - - cuda-command-line-tools {{ cuda_constraints }} - - cuda-compiler {{ cuda_constraints }} + - cuda-libraries {{ cuda_constraints }} + - cuda-nvtx {{ cuda_constraints }} + - libnvjpeg {{ libnvjpeg_constraints }} - cuda-cudart {{ cuda_constraints }} - - cuda-cudart-dev {{ cuda_constraints }} - - cuda-cuobjdump {{ cuda_constraints }} - cuda-cupti {{ cuda_constraints }} - - cuda-cuxxfilt {{ cuda_constraints }} - - cuda-driver-dev {{ cuda_constraints }} - - cuda-libraries {{ cuda_constraints }} - - cuda-libraries-dev {{ cuda_constraints }} - - cuda-cudaart-dev {{ cuda_constraints }} - - cuda-nvcc {{ cuda_constraints }} - - cuda-nvml-dev {{ cuda_constraints }} - - cuda-nvprune {{ cuda_constraints }} - cuda-nvrtc {{ cuda_constraints }} - - cuda-nvrtc-dev {{ cuda_constraints }} - - cuda-nvtx {{ cuda_constraints }} - cuda-runtime {{ cuda_constraints }} - - cuda-toolkit {{ cuda_constraints }} - - cuda-tools {{ cuda_constraints }} - # None, pytorch should depend on pytorch-cuda + - libcufft {{ libcufft_constraints }} + - libcublas {{ libcublas_constraints }} + - libcusolver {{ libcusolver_constraints }} + - libcusparse {{ libcusparse_constraints }} + - libnpp {{ libnpp_constraints }} + - cuda-libraries-dev {{ cuda_constraints }} # [win64] + - libnvjpeg-dev {{ libnvjpeg_constraints }} # [win64] + - cuda-cudart-dev {{ cuda_constraints }} # [win64] + - cuda-nvrtc-dev {{ cuda_constraints }} # [win64] + - libcufft-dev {{ libcufft_constraints }} # [win64] + - libcublas-dev {{ libcublas_constraints }} # [win64] + - libcusolver-dev {{ libcusolver_constraints }} # [win64] + - libcusparse-dev {{ libcusparse_constraints }} # [win64] + - libnpp-dev {{ libnpp_constraints }} # [win64] test: commands: - echo "pytorch-cuda metapackage is created." diff --git a/conda/pytorch-nightly/bld.bat b/conda/pytorch-nightly/bld.bat index 09dc1a8167..18850f758a 100644 --- a/conda/pytorch-nightly/bld.bat +++ b/conda/pytorch-nightly/bld.bat @@ -20,22 +20,18 @@ if "%build_with_cuda%" == "" goto cuda_flags_end set CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v%desired_cuda% set CUDA_BIN_PATH=%CUDA_PATH%\bin set TORCH_NVCC_FLAGS=-Xfatbin -compress-all -set TORCH_CUDA_ARCH_LIST=3.7+PTX;5.0 -if "%desired_cuda%" == "10.2" set TORCH_CUDA_ARCH_LIST=%TORCH_CUDA_ARCH_LIST%;6.0;6.1;7.0;7.5 -if "%desired_cuda%" == "11.3" ( - set TORCH_CUDA_ARCH_LIST=%TORCH_CUDA_ARCH_LIST%;6.0;6.1;7.0;7.5;8.0;8.6 - set TORCH_NVCC_FLAGS=-Xfatbin -compress-all --threads 2 -) +set TORCH_CUDA_ARCH_LIST=3.7+PTX;5.0;6.0;6.1;7.0;7.5;8.0;8.6 if "%desired_cuda%" == "11.5" ( - set TORCH_CUDA_ARCH_LIST=%TORCH_CUDA_ARCH_LIST%;6.0;6.1;7.0;7.5;8.0;8.6 set TORCH_NVCC_FLAGS=-Xfatbin -compress-all --threads 2 ) if "%desired_cuda%" == "11.6" ( - set TORCH_CUDA_ARCH_LIST=%TORCH_CUDA_ARCH_LIST%;6.0;6.1;7.0;7.5;8.0;8.6 set TORCH_NVCC_FLAGS=-Xfatbin -compress-all --threads 2 ) if "%desired_cuda%" == "11.7" ( - set TORCH_CUDA_ARCH_LIST=%TORCH_CUDA_ARCH_LIST%;6.0;6.1;7.0;7.5;8.0;8.6 + set TORCH_NVCC_FLAGS=-Xfatbin -compress-all --threads 2 +) +if "%desired_cuda%" == "11.8" ( + set TORCH_CUDA_ARCH_LIST=%TORCH_CUDA_ARCH_LIST%;9.0 set TORCH_NVCC_FLAGS=-Xfatbin -compress-all --threads 2 ) @@ -112,6 +108,7 @@ IF "%USE_SCCACHE%" == "1" ( if NOT "%build_with_cuda%" == "" ( copy "%CUDA_BIN_PATH%\cudnn*64_*.dll*" %SP_DIR%\torch\lib + copy "%NVTOOLSEXT_PATH%\bin\x64\nvToolsExt64_*.dll*" %SP_DIR%\torch\lib :: cupti library file name changes aggressively, bundle it to avoid :: potential file name mismatch. copy "%CUDA_PATH%\extras\CUPTI\lib64\cupti64_*.dll*" %SP_DIR%\torch\lib diff --git a/conda/pytorch-nightly/build.sh b/conda/pytorch-nightly/build.sh index 05a496fc69..ad1871ac4c 100755 --- a/conda/pytorch-nightly/build.sh +++ b/conda/pytorch-nightly/build.sh @@ -8,6 +8,7 @@ export PYTORCH_BUILD_VERSION=$PKG_VERSION export PYTORCH_BUILD_NUMBER=$PKG_BUILDNUM export USE_LLVM="/opt/llvm_no_cxx11_abi" export LLVM_DIR="$USE_LLVM/lib/cmake/llvm" +export PACKAGE_TYPE="conda" # set OPENSSL_ROOT_DIR=/opt/openssl if it exists if [[ -e /opt/openssl ]]; then @@ -51,41 +52,35 @@ if [[ -z "$USE_CUDA" || "$USE_CUDA" == 1 ]]; then fi if [[ -n "$build_with_cuda" ]]; then export TORCH_NVCC_FLAGS="-Xfatbin -compress-all" - export TORCH_CUDA_ARCH_LIST="3.7+PTX;5.0" + TORCH_CUDA_ARCH_LIST="3.7+PTX;5.0" export USE_STATIC_CUDNN=1 # links cudnn statically (driven by tools/setup_helpers/cudnn.py) - if [[ $CUDA_VERSION == 10* ]]; then - export TORCH_CUDA_ARCH_LIST="$TORCH_CUDA_ARCH_LIST;6.0;6.1;7.0;7.5" - DEPS_LIST=(/usr/local/cuda-10.2/extras/CUPTI/lib64/libcupti.so.10.2) - elif [[ $CUDA_VERSION == 11.3* ]]; then - export TORCH_CUDA_ARCH_LIST="$TORCH_CUDA_ARCH_LIST;6.0;6.1;7.0;7.5;8.0;8.6" - #for cuda 11.3 we use cudnn 8.3.2.44 https://docs.nvidia.com/deeplearning/cudnn/release-notes/rel_8.html - #which does not have single static libcudnn_static.a deliverable to link with - export USE_STATIC_CUDNN=0 - #for cuda 11.3 include all dynamic loading libraries - DEPS_LIST=(/usr/local/cuda/lib64/libcudnn*.so.8 /usr/local/cuda-11.3/extras/CUPTI/lib64/libcupti.so.11.3) - elif [[ $CUDA_VERSION == 11.5* ]]; then - export TORCH_CUDA_ARCH_LIST="$TORCH_CUDA_ARCH_LIST;6.0;6.1;7.0;7.5;8.0;8.6" - #for cuda 11.5 we use cudnn 8.3.2.44 https://docs.nvidia.com/deeplearning/cudnn/release-notes/rel_8.html - #which does not have single static libcudnn_static.a deliverable to link with - export USE_STATIC_CUDNN=0 - #for cuda 11.5 include all dynamic loading libraries - DEPS_LIST=(/usr/local/cuda/lib64/libcudnn*.so.8 /usr/local/cuda-11.5/extras/CUPTI/lib64/libcupti.so.11.5) - elif [[ $CUDA_VERSION == 11.6* ]]; then - export TORCH_CUDA_ARCH_LIST="$TORCH_CUDA_ARCH_LIST;6.0;6.1;7.0;7.5;8.0;8.6" + if [[ $CUDA_VERSION == 11.6* ]]; then + TORCH_CUDA_ARCH_LIST="$TORCH_CUDA_ARCH_LIST;6.0;6.1;7.0;7.5;8.0;8.6" #for cuda 11.5 we use cudnn 8.3.2.44 https://docs.nvidia.com/deeplearning/cudnn/release-notes/rel_8.html #which does not have single static libcudnn_static.a deliverable to link with export USE_STATIC_CUDNN=0 #for cuda 11.5 include all dynamic loading libraries DEPS_LIST=(/usr/local/cuda/lib64/libcudnn*.so.8 /usr/local/cuda-11.6/extras/CUPTI/lib64/libcupti.so.11.6) elif [[ $CUDA_VERSION == 11.7* ]]; then - export TORCH_CUDA_ARCH_LIST="$TORCH_CUDA_ARCH_LIST;6.0;6.1;7.0;7.5;8.0;8.6" + TORCH_CUDA_ARCH_LIST="$TORCH_CUDA_ARCH_LIST;6.0;6.1;7.0;7.5;8.0;8.6" #for cuda 11.7 we use cudnn 8.5 #which does not have single static libcudnn_static.a deliverable to link with export USE_STATIC_CUDNN=0 #for cuda 11.7 include all dynamic loading libraries DEPS_LIST=(/usr/local/cuda/lib64/libcudnn*.so.8 /usr/local/cuda-11.7/extras/CUPTI/lib64/libcupti.so.11.7) + elif [[ $CUDA_VERSION == 11.8* ]]; then + TORCH_CUDA_ARCH_LIST="$TORCH_CUDA_ARCH_LIST;6.0;6.1;7.0;7.5;8.0;8.6;9.0" + #for cuda 11.8 we use cudnn 8.7 + #which does not have single static libcudnn_static.a deliverable to link with + export USE_STATIC_CUDNN=0 + #for cuda 11.8 include all dynamic loading libraries + DEPS_LIST=(/usr/local/cuda/lib64/libcudnn*.so.8 /usr/local/cuda-11.8/extras/CUPTI/lib64/libcupti.so.11.8) + fi + if [[ -n "$OVERRIDE_TORCH_CUDA_ARCH_LIST" ]]; then + TORCH_CUDA_ARCH_LIST="$OVERRIDE_TORCH_CUDA_ARCH_LIST" fi + export TORCH_CUDA_ARCH_LIST="$TORCH_CUDA_ARCH_LIST" export NCCL_ROOT_DIR=/usr/local/cuda export USE_STATIC_NCCL=1 # links nccl statically (driven by tools/setup_helpers/nccl.py, some of the NCCL cmake files such as FindNCCL.cmake and gloo/FindNCCL.cmake) diff --git a/conda/pytorch-nightly/meta.yaml b/conda/pytorch-nightly/meta.yaml index 36e92d5e38..9416a1ed84 100644 --- a/conda/pytorch-nightly/meta.yaml +++ b/conda/pytorch-nightly/meta.yaml @@ -21,15 +21,17 @@ requirements: - pyyaml {% if cross_compile_arm64 == 0 %} - mkl-include # [x86_64] - - mkl=2020.2 # [x86_64 and (not win or py <= 39)] - - mkl=2021.4 # [x86_64 and win and py >= 310] + - mkl=2020.2 # [x86_64 and not win] + - mkl=2021.4 # [x86_64 and win] {% endif %} - typing_extensions - ninja - libuv # [win] - numpy=1.19 # [py <= 39] - - numpy=1.21.5 # [py >= 310] - - openssl=1.1.1l # [py >= 310 and linux] + - numpy=1.21.5 # [py == 310] + - numpy=1.23.5 # [py >= 311] + - openssl=1.1.1l # [py == 310 and linux] + - openssl=1.1.1s # [py >= 311 and linux] {{ environ.get('PYTORCH_LLVM_PACKAGE', ' - llvmdev=9') }} {{ environ.get('MAGMA_PACKAGE', '') }} @@ -41,11 +43,16 @@ requirements: - libuv # [win] - intel-openmp # [win] - typing_extensions + - sympy + - filelock + - networkx + - jinja2 # [py <= 310] {% if cross_compile_arm64 == 0 %} - blas * mkl {% endif %} - pytorch-mutex 1.0 {{ build_variant }} # [not osx ] {{ environ.get('CONDA_CUDATOOLKIT_CONSTRAINT', '') }} +{{ environ.get('CONDA_TRITON_CONSTRAINT', '') }} {% if build_variant == 'cpu' %} run_constrained: @@ -64,6 +71,7 @@ build: - CUDA_VERSION - CUDNN_VERSION - CONDA_CUDATOOLKIT_CONSTRAINT + - CONDA_TRITON_CONSTRAINT - USE_CUDA - CMAKE_ARGS - EXTRA_CAFFE2_CMAKE_FLAGS @@ -81,6 +89,7 @@ build: - USE_PYTORCH_METAL_EXPORT # [osx] - USE_COREML_DELEGATE # [osx] - _GLIBCXX_USE_CXX11_ABI # [unix] + - OVERRIDE_TORCH_CUDA_ARCH_LIST test: imports: diff --git a/conda/torchvision/bld.bat b/conda/torchvision/bld.bat deleted file mode 100644 index 14f6935fba..0000000000 --- a/conda/torchvision/bld.bat +++ /dev/null @@ -1,24 +0,0 @@ -@echo on - -set TORCHVISION_BUILD_VERSION=%PKG_VERSION% -set TORCHVISION_BUILD_NUMBER=%PKG_BUILDNUM% - -if not "%CUDA_VERSION%" == "None" ( - set build_with_cuda=1 - set desired_cuda=%CUDA_VERSION:~0,-1%.%CUDA_VERSION:~-1,1% -) else ( - set build_with_cuda= -) - -if "%build_with_cuda%" == "" goto cuda_flags_end - -set CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v%desired_cuda% -set CUDA_BIN_PATH=%CUDA_PATH%\bin -set NVCC_FLAGS=-D__CUDA_NO_HALF_OPERATORS__ --expt-relaxed-constexpr -if "%desired_cuda%" == "9.0" set NVCC_FLAGS=%NVCC_FLAGS% -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_50,code=compute_50 -if "%desired_cuda%" == "10.0" set NVCC_FLAGS=%NVCC_FLAGS% -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_50,code=compute_50 - -:cuda_flags_end - -python setup.py install --single-version-externally-managed --record=record.txt -if errorlevel 1 exit /b 1 diff --git a/conda/torchvision/meta.yaml b/conda/torchvision/meta.yaml deleted file mode 100644 index caa439c7d2..0000000000 --- a/conda/torchvision/meta.yaml +++ /dev/null @@ -1,53 +0,0 @@ -package: - name: torchvision{{ environ.get('TORCHVISION_PACKAGE_SUFFIX') }} - version: "{{ environ.get('TORCHVISION_BUILD_VERSION') }}" - -source: - git_rev: v{{ environ.get('TORCHVISION_BUILD_VERSION') }} - git_url: https://github.com/pytorch/vision.git - - -requirements: - build: - - {{ compiler('c') }} # [win] - - host: - - python - - setuptools - - pytorch{{ environ.get('TORCHVISION_PACKAGE_SUFFIX') }} >=1.1.0 -{{ environ.get('CONDA_CUDATOOLKIT_CONSTRAINT') }} - - run: - - python - - pillow >=4.1.1 - - numpy >=1.11 - - pytorch{{ environ.get('TORCHVISION_PACKAGE_SUFFIX') }} >=1.1.0 - - six -{{ environ.get('CONDA_CUDATOOLKIT_CONSTRAINT') }} - -build: - number: {{ environ.get('TORCHVISION_BUILD_NUMBER') }} - string: py{{py}}_cu{{ environ['CUDA_VERSION'] }}_{{environ.get('TORCHVISION_BUILD_NUMBER')}} - script: python setup.py install --single-version-externally-managed --record=record.txt # [not win] - script_env: - - CUDA_VERSION - -test: - imports: - - torchvision - - torchvision.datasets - - torchvision.transforms - source_files: - - test - requires: - - pytest - - scipy - commands: - pytest . - - -about: - home: https://github.com/pytorch/vision - license: BSD - license_file: LICENSE - summary: 'image and video datasets and models for torch deep learning' diff --git a/conda/vs2017/conda_build_config.yaml b/conda/vs2017/conda_build_config.yaml deleted file mode 100755 index 5188bb0ebe..0000000000 --- a/conda/vs2017/conda_build_config.yaml +++ /dev/null @@ -1,24 +0,0 @@ -blas_impl: - - mkl # [x86_64] -c_compiler: - - vs2017 # [win] -cxx_compiler: - - vs2017 # [win] -python: - - 3.5 - - 3.6 -# This differs from target_platform in that it determines what subdir the compiler -# will target, not what subdir the compiler package will be itself. -# For example, we need a win-64 vs2008_win-32 package, so that we compile win-32 -# code on win-64 miniconda. -cross_compiler_target_platform: - - win-64 # [win] -target_platform: - - win-64 # [win] -vc: - - 14 -zip_keys: - - # [win] - - vc # [win] - - c_compiler # [win] - - cxx_compiler # [win] diff --git a/conda/vs2017/activate.bat b/conda/vs2022/activate.bat old mode 100755 new mode 100644 similarity index 57% rename from conda/vs2017/activate.bat rename to conda/vs2022/activate.bat index ccecfc2544..fe18f77230 --- a/conda/vs2017/activate.bat +++ b/conda/vs2022/activate.bat @@ -1,19 +1,26 @@ :: Set env vars that tell distutils to use the compiler that we put on path -SET DISTUTILS_USE_SDK=1 -SET MSSdk=1 +set DISTUTILS_USE_SDK=1 +set MSSdk=1 -SET "VS_VERSION=15.0" -SET "VS_MAJOR=15" -SET "VS_YEAR=2017" +set "VS_VERSION=17.4" +set "VS_MAJOR=17" +set "VC_YEAR=2022" +set "VC_VERSION_LOWER=17" +set "VC_VERSION_UPPER=18" set "MSYS2_ARG_CONV_EXCL=/AI;/AL;/OUT;/out" set "MSYS2_ENV_CONV_EXCL=CL" :: For Python 3.5+, ensure that we link with the dynamic runtime. See :: http://stevedower.id.au/blog/building-for-python-3-5-part-two/ for more info -set "PY_VCRUNTIME_REDIST=%PREFIX%\\bin\\vcruntime140.dll" +set "PY_VCRUNTIME_REDIST=%PREFIX%\\bin\\vcruntime143.dll" -for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -products * -version [15^,16^) -property installationPath`) do ( +if not "%VS15INSTALLDIR%" == "" if exist "%VS15INSTALLDIR%\VC\Auxiliary\Build\vcvarsall.bat" ( + set "VSINSTALLDIR=%VS15INSTALLDIR%\" + goto :vswhere +) + +for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -products * -version [%VC_VERSION_LOWER%^,%VC_VERSION_UPPER%^) -property installationPath`) do ( if exist "%%i" if exist "%%i\VC\Auxiliary\Build\vcvarsall.bat" ( set "VSINSTALLDIR=%%i\" goto :vswhere @@ -23,15 +30,15 @@ for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio :vswhere :: Shorten PATH to avoid the `input line too long` error. -SET MyPath=%PATH% +set MyPath=%PATH% setlocal EnableDelayedExpansion -SET TempPath="%MyPath:;=";"%" -SET var= -FOR %%a IN (%TempPath%) DO ( - IF EXIST %%~sa ( - SET "var=!var!;%%~sa" +set TempPath="%MyPath:;=";"%" +set var= +for %%a in (%TempPath%) do ( + if exist %%~sa ( + set "var=!var!;%%~sa" ) ) @@ -39,6 +46,6 @@ set "TempPath=!var:~1!" endlocal & set "PATH=%TempPath%" :: Shorten current directory too -FOR %%A IN (.) DO CD "%%~sA" +for %%A in (.) do cd "%%~sA" :: other things added by install_activate.bat at package build time diff --git a/conda/torchvision/conda_build_config.yaml b/conda/vs2022/conda_build_config.yaml similarity index 86% rename from conda/torchvision/conda_build_config.yaml rename to conda/vs2022/conda_build_config.yaml index 5188bb0ebe..e2a4de3c2e 100644 --- a/conda/torchvision/conda_build_config.yaml +++ b/conda/vs2022/conda_build_config.yaml @@ -1,12 +1,13 @@ blas_impl: - mkl # [x86_64] c_compiler: - - vs2017 # [win] + - vs2022 # [win] cxx_compiler: - - vs2017 # [win] + - vs2022 # [win] python: - - 3.5 - - 3.6 + - 3.8 + - 3.9 + - 3.10 # This differs from target_platform in that it determines what subdir the compiler # will target, not what subdir the compiler package will be itself. # For example, we need a win-64 vs2008_win-32 package, so that we compile win-32 diff --git a/conda/vs2017/install_activate.bat b/conda/vs2022/install_activate.bat old mode 100755 new mode 100644 similarity index 98% rename from conda/vs2017/install_activate.bat rename to conda/vs2022/install_activate.bat index 2ca223ebc8..eb85767d67 --- a/conda/vs2017/install_activate.bat +++ b/conda/vs2022/install_activate.bat @@ -1,5 +1,5 @@ -set YEAR=2017 -set VER=15 +set YEAR=2022 +set VER=17 mkdir "%PREFIX%\etc\conda\activate.d" copy "%RECIPE_DIR%\activate.bat" "%PREFIX%\etc\conda\activate.d\vs%YEAR%_compiler_vars.bat" diff --git a/conda/vs2017/install_runtime.bat b/conda/vs2022/install_runtime.bat old mode 100755 new mode 100644 similarity index 92% rename from conda/vs2017/install_runtime.bat rename to conda/vs2022/install_runtime.bat index 5163c16cf2..bac684dae6 --- a/conda/vs2017/install_runtime.bat +++ b/conda/vs2022/install_runtime.bat @@ -3,7 +3,7 @@ if "%ARCH%"=="64" ( set VC_PATH=x64 ) -set MSC_VER=2017 +set MSC_VER=2022 rem :: This should always be present for VC installed with VS. Not sure about VC installed with Visual C++ Build Tools 2015 rem FOR /F "usebackq tokens=3*" %%A IN (`REG QUERY "HKEY_LOCAL_MACHINE\Software\Microsoft\DevDiv\VC\Servicing\14.0\IDE.x64" /v UpdateVersion`) DO ( @@ -23,10 +23,10 @@ robocopy "C:\Program Files (x86)\Windows Kits\10\Redist\ucrt\DLLs\%VC_PATH%" "% robocopy "C:\Program Files (x86)\Windows Kits\10\Redist\ucrt\DLLs\%VC_PATH%" "%PREFIX%" *.dll /E if %ERRORLEVEL% GEQ 8 exit 1 -REM ========== This one comes from visual studio 2017 -set "VC_VER=141" +REM ========== This one comes from visual studio 2022 +set "VC_VER=143" -for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -products * -version [15^,16^) -property installationPath`) do ( +for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -products * -version [17^,18^) -property installationPath`) do ( if exist "%%i" if exist "%%i\VC\Auxiliary\Build\vcvarsall.bat" ( set "VS15VCVARSALL=%%i\VC\Auxiliary\Build\vcvarsall.bat" goto :eof diff --git a/conda/vs2017/meta.yaml b/conda/vs2022/meta.yaml old mode 100755 new mode 100644 similarity index 66% rename from conda/vs2017/meta.yaml rename to conda/vs2022/meta.yaml index 1f569525ee..184c4c32df --- a/conda/vs2017/meta.yaml +++ b/conda/vs2022/meta.yaml @@ -1,7 +1,7 @@ -{% set vcver="14.1" %} -{% set vcfeature="14" %} -{% set vsyear="2017" %} -{% set fullver="15.4.27004.2010" %} +{% set vcver="17.4" %} +{% set vcfeature="17" %} +{% set vsyear="2022" %} +{% set fullver="17.4.33110.190" %} package: name: vs{{ vsyear }} @@ -16,7 +16,7 @@ outputs: - name: vs{{ vsyear }}_{{ cross_compiler_target_platform }} script: install_activate.bat track_features: - # VS 2017 is binary-compatible with VS 2015/vc14. Tools are "v141". + # VS 2022 is binary-compatible with VS 2019/vc 14.2, VS 2017/vc 14.1 and 2015/vc14. Tools are "v143". strong: - vc{{ vcfeature }} about: diff --git a/cron/nightly_defaults.sh b/cron/nightly_defaults.sh index 0f6532adc1..d8b6f5ee04 100755 --- a/cron/nightly_defaults.sh +++ b/cron/nightly_defaults.sh @@ -120,7 +120,7 @@ if [[ ! -d "$NIGHTLIES_PYTORCH_ROOT" ]]; then export PYTORCH_BRANCH="$last_commit" fi git checkout "$PYTORCH_BRANCH" - git submodule update --jobs 0 + git submodule update popd fi @@ -229,7 +229,7 @@ if [[ "$DAYS_TO_KEEP" < '1' ]]; then fi # PYTORCH_NIGHTLIES_TIMEOUT -# Timeout in seconds. +# Timeout in seconds. # When full testing is enabled, condas builds often take up to 2 hours 20 # minutes, so the default is set to (2 * 60 + 20 + 40 [buffer]) * 60 == 10800 # seconds. diff --git a/libtorch/Dockerfile b/libtorch/Dockerfile index f38aca0632..c5eb904ce6 100644 --- a/libtorch/Dockerfile +++ b/libtorch/Dockerfile @@ -42,19 +42,7 @@ ENV CUDA_HOME /usr/local/cuda FROM base as conda ADD ./common/install_conda.sh install_conda.sh RUN bash ./install_conda.sh && rm install_conda.sh -RUN /opt/conda/bin/conda install -y cmake=3.14 - -FROM cuda as cuda10.2 -RUN bash ./install_cuda.sh 10.2 -RUN bash ./install_magma.sh 10.2 - -FROM cuda as cuda11.3 -RUN bash ./install_cuda.sh 11.3 -RUN bash ./install_magma.sh 11.3 - -FROM cuda as cuda11.5 -RUN bash ./install_cuda.sh 11.5 -RUN bash ./install_magma.sh 11.5 +RUN /opt/conda/bin/conda install -y cmake=3.18 FROM cuda as cuda11.6 RUN bash ./install_cuda.sh 11.6 @@ -64,6 +52,10 @@ FROM cuda as cuda11.7 RUN bash ./install_cuda.sh 11.7 RUN bash ./install_magma.sh 11.7 +FROM cuda as cuda11.8 +RUN bash ./install_cuda.sh 11.8 +RUN bash ./install_magma.sh 11.8 + FROM cpu as rocm ARG PYTORCH_ROCM_ARCH ENV PYTORCH_ROCM_ARCH ${PYTORCH_ROCM_ARCH} @@ -77,23 +69,13 @@ RUN apt-get update -y && \ apt-get install python -y && \ apt-get clean -FROM rocm as rocm5.2 -RUN ROCM_VERSION=5.2 bash ./install_rocm.sh && rm install_rocm.sh -RUN bash ./install_rocm_drm.sh && rm install_rocm_drm.sh -RUN bash ./install_rocm_magma.sh && rm install_rocm_magma.sh - -FROM rocm as rocm5.1.3 -RUN ROCM_VERSION=5.1.3 bash ./install_rocm.sh && rm install_rocm.sh -RUN bash ./install_rocm_drm.sh && rm install_rocm_drm.sh -RUN bash ./install_rocm_magma.sh && rm install_rocm_magma.sh - -FROM rocm as rocm5.1.1 -RUN ROCM_VERSION=5.1.1 bash ./install_rocm.sh && rm install_rocm.sh +FROM rocm as rocm5.4.2 +RUN ROCM_VERSION=5.4.2 bash ./install_rocm.sh && rm install_rocm.sh RUN bash ./install_rocm_drm.sh && rm install_rocm_drm.sh RUN bash ./install_rocm_magma.sh && rm install_rocm_magma.sh -FROM rocm as rocm5.1 -RUN ROCM_VERSION=5.1 bash ./install_rocm.sh && rm install_rocm.sh +FROM rocm as rocm5.3 +RUN ROCM_VERSION=5.3 bash ./install_rocm.sh && rm install_rocm.sh RUN bash ./install_rocm_drm.sh && rm install_rocm_drm.sh RUN bash ./install_rocm_magma.sh && rm install_rocm_magma.sh diff --git a/libtorch/build.sh b/libtorch/build.sh index b2551a6be3..88c8c6f9a8 100644 --- a/libtorch/build.sh +++ b/libtorch/build.sh @@ -7,4 +7,4 @@ set -ex SCRIPTPATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" -BUILD_PYTHONLESS=1 DESIRED_PYTHON="3.7" ${SCRIPTPATH}/../manywheel/build.sh +BUILD_PYTHONLESS=1 DESIRED_PYTHON="3.8" ${SCRIPTPATH}/../manywheel/build.sh diff --git a/libtorch/build_all_docker.sh b/libtorch/build_all_docker.sh index 5703ca41cc..8d25da9bcd 100755 --- a/libtorch/build_all_docker.sh +++ b/libtorch/build_all_docker.sh @@ -4,10 +4,10 @@ set -eou pipefail TOPDIR=$(git rev-parse --show-toplevel) -for cuda_version in 11.7 11.6 11.5 11.3 10.2; do +for cuda_version in 11.8 11.7 11.6; do GPU_ARCH_TYPE=cuda GPU_ARCH_VERSION="${cuda_version}" "${TOPDIR}/libtorch/build_docker.sh" done -for rocm_version in 5.1.1 5.2; do +for rocm_version in 5.3 5.4.2; do GPU_ARCH_TYPE=rocm GPU_ARCH_VERSION="${rocm_version}" "${TOPDIR}/libtorch/build_docker.sh" done diff --git a/libtorch/build_docker.sh b/libtorch/build_docker.sh index fe441bb9a6..bbf42b1d02 100755 --- a/libtorch/build_docker.sh +++ b/libtorch/build_docker.sh @@ -27,7 +27,7 @@ case ${GPU_ARCH_TYPE} in rocm) BASE_TARGET=rocm${GPU_ARCH_VERSION} DOCKER_TAG=rocm${GPU_ARCH_VERSION} - GPU_IMAGE=rocm/dev-ubuntu-18.04:${GPU_ARCH_VERSION} + GPU_IMAGE=rocm/dev-ubuntu-20.04:${GPU_ARCH_VERSION} PYTORCH_ROCM_ARCH="gfx900;gfx906;gfx908" ROCM_REGEX="([0-9]+)\.([0-9]+)[\.]?([0-9]*)" if [[ $GPU_ARCH_VERSION =~ $ROCM_REGEX ]]; then diff --git a/magma/Makefile b/magma/Makefile index 2d690c14c8..4a90a43e2b 100644 --- a/magma/Makefile +++ b/magma/Makefile @@ -1,8 +1,8 @@ SHELL=/usr/bin/env bash -DESIRED_CUDA ?= 11.3 -PACKAGE_NAME ?= magma-cuda113 -CUDA_ARCH_LIST ?= -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 +DESIRED_CUDA ?= 11.6 +PACKAGE_NAME ?= magma-cuda116 +CUDA_ARCH_LIST ?= -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 DOCKER_RUN = set -eou pipefail; docker run --rm -i \ -v $(shell git rev-parse --show-toplevel):/builder \ @@ -14,47 +14,30 @@ DOCKER_RUN = set -eou pipefail; docker run --rm -i \ magma/build_magma.sh .PHONY: all +all: magma-cuda118 all: magma-cuda117 all: magma-cuda116 -all: magma-cuda115 -all: magma-cuda113 -all: magma-cuda102 .PHONY: clean: $(RM) -r magma-* $(RM) -r output +.PHONY: magma-cuda118 +magma-cuda118: DESIRED_CUDA := 11.8 +magma-cuda118: PACKAGE_NAME := magma-cuda118 +magma-cuda118: CUDA_ARCH_LIST += -gencode arch=compute_90,code=sm_90 +magma-cuda118: + $(DOCKER_RUN) + .PHONY: magma-cuda117 magma-cuda117: DESIRED_CUDA := 11.7 magma-cuda117: PACKAGE_NAME := magma-cuda117 -magma-cuda117: CUDA_ARCH_LIST += -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 magma-cuda117: $(DOCKER_RUN) .PHONY: magma-cuda116 magma-cuda116: DESIRED_CUDA := 11.6 magma-cuda116: PACKAGE_NAME := magma-cuda116 -magma-cuda116: CUDA_ARCH_LIST += -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 magma-cuda116: $(DOCKER_RUN) - -.PHONY: magma-cuda115 -magma-cuda115: DESIRED_CUDA := 11.5 -magma-cuda115: PACKAGE_NAME := magma-cuda115 -magma-cuda115: CUDA_ARCH_LIST += -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -magma-cuda115: - $(DOCKER_RUN) - -.PHONY: magma-cuda113 -magma-cuda113: DESIRED_CUDA := 11.3 -magma-cuda113: PACKAGE_NAME := magma-cuda113 -magma-cuda113: CUDA_ARCH_LIST += -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -magma-cuda113: - $(DOCKER_RUN) - -.PHONY: magma-cuda102 -magma-cuda102: DESIRED_CUDA := 10.2 -magma-cuda102: PACKAGE_NAME := magma-cuda102 -magma-cuda102: - $(DOCKER_RUN) diff --git a/manywheel/Dockerfile b/manywheel/Dockerfile index 3140f98638..43a7d0568b 100644 --- a/manywheel/Dockerfile +++ b/manywheel/Dockerfile @@ -21,9 +21,10 @@ RUN wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm & rpm -ivh epel-release-latest-7.noarch.rpm && \ rm -f epel-release-latest-7.noarch.rpm -# cmake -RUN yum install -y cmake3 && \ - ln -s /usr/bin/cmake3 /usr/bin/cmake +# cmake-3.18.4 from pip +RUN yum install -y python3-pip && \ + python3 -mpip install cmake==3.18.4 && \ + ln -s /usr/local/bin/cmake /usr/bin/cmake RUN yum install -y autoconf aclocal automake make @@ -35,18 +36,19 @@ FROM base as openssl ADD ./common/install_openssl.sh install_openssl.sh RUN bash ./install_openssl.sh && rm install_openssl.sh -FROM base as python +# EPEL for cmake +FROM base as patchelf +# Install patchelf +ADD ./common/install_patchelf.sh install_patchelf.sh +RUN bash ./install_patchelf.sh && rm install_patchelf.sh +RUN cp $(which patchelf) /patchelf + +FROM patchelf as python # build python COPY manywheel/build_scripts /build_scripts ADD ./common/install_cpython.sh /build_scripts/install_cpython.sh RUN bash build_scripts/build.sh && rm -r build_scripts -# remove unncessary python versions -RUN rm -rf /opt/python/cp26-cp26m /opt/_internal/cpython-2.6.9-ucs2 -RUN rm -rf /opt/python/cp26-cp26mu /opt/_internal/cpython-2.6.9-ucs4 -RUN rm -rf /opt/python/cp33-cp33m /opt/_internal/cpython-3.3.6 -RUN rm -rf /opt/python/cp34-cp34m /opt/_internal/cpython-3.4.6 - FROM base as cuda ARG BASE_CUDA_VERSION=10.2 # Install CUDA @@ -58,13 +60,6 @@ FROM base as intel ADD ./common/install_mkl.sh install_mkl.sh RUN bash ./install_mkl.sh && rm install_mkl.sh -# EPEL for cmake -FROM base as patchelf -# Install patchelf -ADD ./common/install_patchelf.sh install_patchelf.sh -RUN bash ./install_patchelf.sh && rm install_patchelf.sh -RUN cp $(which patchelf) /patchelf - FROM base as magma ARG BASE_CUDA_VERSION=10.2 # Install magma @@ -142,9 +137,12 @@ RUN yum install -y devtoolset-${DEVTOOLSET_VERSION}-gcc devtoolset-${DEVTOOLSET_ ENV PATH=/opt/rh/devtoolset-${DEVTOOLSET_VERSION}/root/usr/bin:$PATH ENV LD_LIBRARY_PATH=/opt/rh/devtoolset-${DEVTOOLSET_VERSION}/root/usr/lib64:/opt/rh/devtoolset-${DEVTOOLSET_VERSION}/root/usr/lib:$LD_LIBRARY_PATH -# cmake -RUN yum install -y cmake3 && \ - ln -s /usr/bin/cmake3 /usr/bin/cmake +# cmake is already installed inside the rocm base image, so remove if present +RUN rpm -e cmake || true +# cmake-3.18.4 from pip +RUN yum install -y python3-pip && \ + python3 -mpip install cmake==3.18.4 && \ + ln -s /usr/local/bin/cmake /usr/bin/cmake # ninja RUN yum install -y http://repo.okay.com.mx/centos/7/x86_64/release/okay-release-1-5.el7.noarch.rpm @@ -155,7 +153,7 @@ RUN rm -rf /usr/local/cuda-${BASE_CUDA_VERSION} COPY --from=cuda /usr/local/cuda-${BASE_CUDA_VERSION} /usr/local/cuda-${BASE_CUDA_VERSION} COPY --from=magma /usr/local/cuda-${BASE_CUDA_VERSION} /usr/local/cuda-${BASE_CUDA_VERSION} -FROM common as rocm_final +FROM cpu_final as rocm_final ARG ROCM_VERSION=3.7 ARG PYTORCH_ROCM_ARCH ENV PYTORCH_ROCM_ARCH ${PYTORCH_ROCM_ARCH} @@ -166,3 +164,5 @@ ADD ./common/install_rocm_drm.sh install_rocm_drm.sh RUN bash ./install_rocm_drm.sh && rm install_rocm_drm.sh ADD ./common/install_rocm_magma.sh install_rocm_magma.sh RUN bash ./install_rocm_magma.sh && rm install_rocm_magma.sh +# cmake3 is needed for the MIOpen build +RUN ln -sf /usr/local/bin/cmake /usr/bin/cmake3 diff --git a/manywheel/Dockerfile_cxx11-abi b/manywheel/Dockerfile_cxx11-abi index 966d570869..a5b0673e97 100644 --- a/manywheel/Dockerfile_cxx11-abi +++ b/manywheel/Dockerfile_cxx11-abi @@ -13,7 +13,6 @@ RUN yum -y update RUN yum install -y wget curl perl util-linux xz bzip2 git patch which zlib-devel RUN yum install -y autoconf automake make cmake gdb gcc gcc-c++ - FROM base as openssl ADD ./common/install_openssl.sh install_openssl.sh RUN bash ./install_openssl.sh && rm install_openssl.sh diff --git a/manywheel/build_all_docker.sh b/manywheel/build_all_docker.sh index d50eea49d7..395f71be36 100644 --- a/manywheel/build_all_docker.sh +++ b/manywheel/build_all_docker.sh @@ -9,12 +9,12 @@ MANYLINUX_VERSION=2014 GPU_ARCH_TYPE=cpu "${TOPDIR}/manywheel/build_docker.sh" GPU_ARCH_TYPE=cpu-cxx11-abi "${TOPDIR}/manywheel/build_docker.sh" -for cuda_version in 11.5 11.3 10.2; do +for cuda_version in 11.7 11.6; do GPU_ARCH_TYPE=cuda GPU_ARCH_VERSION="${cuda_version}" "${TOPDIR}/manywheel/build_docker.sh" MANYLINUX_VERSION=2014 GPU_ARCH_TYPE=cuda GPU_ARCH_VERSION="${cuda_version}" "${TOPDIR}/manywheel/build_docker.sh" done -for rocm_version in 5.1.1 5.2; do +for rocm_version in 5.3 5.4.2; do GPU_ARCH_TYPE=rocm GPU_ARCH_VERSION="${rocm_version}" "${TOPDIR}/manywheel/build_docker.sh" MANYLINUX_VERSION=2014 GPU_ARCH_TYPE=rocm GPU_ARCH_VERSION="${rocm_version}" "${TOPDIR}/manywheel/build_docker.sh" done diff --git a/manywheel/build_common.sh b/manywheel/build_common.sh index 878d81628b..cc56e695a4 100644 --- a/manywheel/build_common.sh +++ b/manywheel/build_common.sh @@ -371,15 +371,15 @@ for pkg in /$WHEELHOUSE_DIR/torch*linux*.whl /$LIBTORCH_HOUSE_DIR/libtorch*.zip; # set RPATH of _C.so and similar to $ORIGIN, $ORIGIN/lib find $PREFIX -maxdepth 1 -type f -name "*.so*" | while read sofile; do - echo "Setting rpath of $sofile to " '$ORIGIN:$ORIGIN/lib' - $PATCHELF_BIN --set-rpath '$ORIGIN:$ORIGIN/lib' $sofile + echo "Setting rpath of $sofile to ${C_SO_RPATH:-'$ORIGIN:$ORIGIN/lib'}" + $PATCHELF_BIN --set-rpath ${C_SO_RPATH:-'$ORIGIN:$ORIGIN/lib'} ${FORCE_RPATH:-} $sofile $PATCHELF_BIN --print-rpath $sofile done # set RPATH of lib/ files to $ORIGIN find $PREFIX/lib -maxdepth 1 -type f -name "*.so*" | while read sofile; do - echo "Setting rpath of $sofile to " '$ORIGIN' - $PATCHELF_BIN --set-rpath '$ORIGIN' $sofile + echo "Setting rpath of $sofile to ${LIB_SO_RPATH:-'$ORIGIN'}" + $PATCHELF_BIN --set-rpath ${LIB_SO_RPATH:-'$ORIGIN'} ${FORCE_RPATH:-} $sofile $PATCHELF_BIN --print-rpath $sofile done @@ -387,10 +387,10 @@ for pkg in /$WHEELHOUSE_DIR/torch*linux*.whl /$LIBTORCH_HOUSE_DIR/libtorch*.zip; record_file=$(echo $(basename $pkg) | sed -e 's/-cp.*$/.dist-info\/RECORD/g') if [[ -e $record_file ]]; then echo "Generating new record file $record_file" - rm -f $record_file + : > "$record_file" # generate records for folders in wheel find * -type f | while read fname; do - echo $(make_wheel_record $fname) >>$record_file + make_wheel_record "$fname" >>"$record_file" done fi diff --git a/manywheel/build_cuda.sh b/manywheel/build_cuda.sh index efea1ae93d..6b5cd91117 100644 --- a/manywheel/build_cuda.sh +++ b/manywheel/build_cuda.sh @@ -58,12 +58,12 @@ cuda_version_nodot=$(echo $CUDA_VERSION | tr -d '.') TORCH_CUDA_ARCH_LIST="3.7;5.0;6.0;7.0" case ${CUDA_VERSION} in - 11.[3567]) - TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST};7.5;8.0;8.6" + 11.8) + TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST};7.5;8.0;8.6;9.0" EXTRA_CAFFE2_CMAKE_FLAGS+=("-DATEN_NO_TEST=ON") ;; - 10.*) - TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST}" + 11.[67]) + TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST};7.5;8.0;8.6" EXTRA_CAFFE2_CMAKE_FLAGS+=("-DATEN_NO_TEST=ON") ;; *) @@ -108,96 +108,7 @@ elif [[ "$OS_NAME" == *"Ubuntu"* ]]; then LIBGOMP_PATH="/usr/lib/x86_64-linux-gnu/libgomp.so.1" fi -if [[ $CUDA_VERSION == "10.2" ]]; then -DEPS_LIST=( - "/usr/local/cuda/lib64/libcudart.so.10.2" - "/usr/local/cuda/lib64/libnvToolsExt.so.1" - "/usr/local/cuda/lib64/libnvrtc.so.10.2" - "/usr/local/cuda/lib64/libnvrtc-builtins.so" - "/usr/local/cuda/lib64/libcublas.so.10" - "/usr/local/cuda/lib64/libcublasLt.so.10" - "$LIBGOMP_PATH" -) - -DEPS_SONAME=( - "libcudart.so.10.2" - "libnvToolsExt.so.1" - "libnvrtc.so.10.2" - "libnvrtc-builtins.so" - "libcublas.so.10" - "libcublasLt.so.10" - "libgomp.so.1" -) -elif [[ $CUDA_VERSION == "11.3" ]]; then -export USE_STATIC_CUDNN=0 -DEPS_LIST=( - "/usr/local/cuda/lib64/libcudart.so.11.0" - "/usr/local/cuda/lib64/libnvToolsExt.so.1" - "/usr/local/cuda/lib64/libnvrtc.so.11.2" # this is not a mistake for 11.3, it links to 11.3.58 - "/usr/local/cuda/lib64/libnvrtc-builtins.so.11.3" - "/usr/local/cuda/lib64/libcudnn_adv_infer.so.8" - "/usr/local/cuda/lib64/libcudnn_adv_train.so.8" - "/usr/local/cuda/lib64/libcudnn_cnn_infer.so.8" - "/usr/local/cuda/lib64/libcudnn_cnn_train.so.8" - "/usr/local/cuda/lib64/libcudnn_ops_infer.so.8" - "/usr/local/cuda/lib64/libcudnn_ops_train.so.8" - "/usr/local/cuda/lib64/libcudnn.so.8" - "/usr/local/cuda/lib64/libcublas.so.11" - "/usr/local/cuda/lib64/libcublasLt.so.11" - "$LIBGOMP_PATH" -) - -DEPS_SONAME=( - "libcudart.so.11.0" - "libnvToolsExt.so.1" - "libnvrtc.so.11.2" - "libnvrtc-builtins.so.11.3" - "libcudnn_adv_infer.so.8" - "libcudnn_adv_train.so.8" - "libcudnn_cnn_infer.so.8" - "libcudnn_cnn_train.so.8" - "libcudnn_ops_infer.so.8" - "libcudnn_ops_train.so.8" - "libcudnn.so.8" - "libcublas.so.11" - "libcublasLt.so.11" - "libgomp.so.1" -) -elif [[ $CUDA_VERSION == "11.5" ]]; then -export USE_STATIC_CUDNN=0 -DEPS_LIST=( - "/usr/local/cuda/lib64/libcudart.so.11.0" - "/usr/local/cuda/lib64/libnvToolsExt.so.1" - "/usr/local/cuda/lib64/libnvrtc.so.11.2" # this is not a mistake for 11.5, it links to 11.5.50 - "/usr/local/cuda/lib64/libnvrtc-builtins.so.11.5" - "/usr/local/cuda/lib64/libcudnn_adv_infer.so.8" - "/usr/local/cuda/lib64/libcudnn_adv_train.so.8" - "/usr/local/cuda/lib64/libcudnn_cnn_infer.so.8" - "/usr/local/cuda/lib64/libcudnn_cnn_train.so.8" - "/usr/local/cuda/lib64/libcudnn_ops_infer.so.8" - "/usr/local/cuda/lib64/libcudnn_ops_train.so.8" - "/usr/local/cuda/lib64/libcudnn.so.8" - "/usr/local/cuda/lib64/libcublas.so.11" - "/usr/local/cuda/lib64/libcublasLt.so.11" - "$LIBGOMP_PATH" -) -DEPS_SONAME=( - "libcudart.so.11.0" - "libnvToolsExt.so.1" - "libnvrtc.so.11.2" - "libnvrtc-builtins.so.11.5" - "libcudnn_adv_infer.so.8" - "libcudnn_adv_train.so.8" - "libcudnn_cnn_infer.so.8" - "libcudnn_cnn_train.so.8" - "libcudnn_ops_infer.so.8" - "libcudnn_ops_train.so.8" - "libcudnn.so.8" - "libcublas.so.11" - "libcublasLt.so.11" - "libgomp.so.1" -) -elif [[ $CUDA_VERSION == "11.6" ]]; then +if [[ $CUDA_VERSION == "11.6" ]]; then export USE_STATIC_CUDNN=0 DEPS_LIST=( "/usr/local/cuda/lib64/libcudart.so.11.0" @@ -231,48 +142,101 @@ DEPS_SONAME=( "libcublasLt.so.11" "libgomp.so.1" ) -elif [[ $CUDA_VERSION == "11.7" ]]; then -export USE_STATIC_CUDNN=0 -DEPS_LIST=( - "/usr/local/cuda/lib64/libcudart.so.11.0" - "/usr/local/cuda/lib64/libnvToolsExt.so.1" - "/usr/local/cuda/lib64/libnvrtc.so.11.2" # this is not a mistake for 11.7, it links to 11.7.50 - "/usr/local/cuda/lib64/libnvrtc-builtins.so.11.7" - "/usr/local/cuda/lib64/libcudnn_adv_infer.so.8" - "/usr/local/cuda/lib64/libcudnn_adv_train.so.8" - "/usr/local/cuda/lib64/libcudnn_cnn_infer.so.8" - "/usr/local/cuda/lib64/libcudnn_cnn_train.so.8" - "/usr/local/cuda/lib64/libcudnn_ops_infer.so.8" - "/usr/local/cuda/lib64/libcudnn_ops_train.so.8" - "/usr/local/cuda/lib64/libcudnn.so.8" - "/usr/local/cuda/lib64/libcublas.so.11" - "/usr/local/cuda/lib64/libcublasLt.so.11" - "$LIBGOMP_PATH" -) -DEPS_SONAME=( - "libcudart.so.11.0" - "libnvToolsExt.so.1" - "libnvrtc.so.11.2" - "libnvrtc-builtins.so.11.7" - "libcudnn_adv_infer.so.8" - "libcudnn_adv_train.so.8" - "libcudnn_cnn_infer.so.8" - "libcudnn_cnn_train.so.8" - "libcudnn_ops_infer.so.8" - "libcudnn_ops_train.so.8" - "libcudnn.so.8" - "libcublas.so.11" - "libcublasLt.so.11" - "libgomp.so.1" -) +elif [[ $CUDA_VERSION == "11.7" || $CUDA_VERSION == "11.8" ]]; then + export USE_STATIC_CUDNN=0 + # Try parallelizing nvcc as well + export TORCH_NVCC_FLAGS="-Xfatbin -compress-all --threads 2" + DEPS_LIST=( + "$LIBGOMP_PATH" + ) + DEPS_SONAME=( + "libgomp.so.1" + ) -# Try parallelizing nvcc as well -export TORCH_NVCC_FLAGS="-Xfatbin -compress-all --threads 2" + if [[ -z "$PYTORCH_EXTRA_INSTALL_REQUIREMENTS" ]]; then + echo "Bundling with cudnn and cublas." + DEPS_LIST+=( + "/usr/local/cuda/lib64/libcudnn_adv_infer.so.8" + "/usr/local/cuda/lib64/libcudnn_adv_train.so.8" + "/usr/local/cuda/lib64/libcudnn_cnn_infer.so.8" + "/usr/local/cuda/lib64/libcudnn_cnn_train.so.8" + "/usr/local/cuda/lib64/libcudnn_ops_infer.so.8" + "/usr/local/cuda/lib64/libcudnn_ops_train.so.8" + "/usr/local/cuda/lib64/libcudnn.so.8" + "/usr/local/cuda/lib64/libcublas.so.11" + "/usr/local/cuda/lib64/libcublasLt.so.11" + "/usr/local/cuda/lib64/libcudart.so.11.0" + "/usr/local/cuda/lib64/libnvToolsExt.so.1" + "/usr/local/cuda/lib64/libnvrtc.so.11.2" # this is not a mistake, it links to more specific cuda version + ) + DEPS_SONAME+=( + "libcudnn_adv_infer.so.8" + "libcudnn_adv_train.so.8" + "libcudnn_cnn_infer.so.8" + "libcudnn_cnn_train.so.8" + "libcudnn_ops_infer.so.8" + "libcudnn_ops_train.so.8" + "libcudnn.so.8" + "libcublas.so.11" + "libcublasLt.so.11" + "libcudart.so.11.0" + "libnvToolsExt.so.1" + "libnvrtc.so.11.2" + ) + if [[ $CUDA_VERSION == "11.7" ]]; then + DEPS_LIST+=( + "/usr/local/cuda/lib64/libnvrtc-builtins.so.11.7" + ) + DEPS_SONAME+=( + "libnvrtc-builtins.so.11.7" + ) + fi + if [[ $CUDA_VERSION == "11.8" ]]; then + DEPS_LIST+=( + "/usr/local/cuda/lib64/libnvrtc-builtins.so.11.8" + ) + DEPS_SONAME+=( + "libnvrtc-builtins.so.11.8" + ) + fi + else + echo "Using nvidia libs from pypi." + CUDA_RPATHS=( + '$ORIGIN/../../nvidia/cublas/lib' + '$ORIGIN/../../nvidia/cuda_cupti/lib' + '$ORIGIN/../../nvidia/cuda_nvrtc/lib' + '$ORIGIN/../../nvidia/cuda_runtime/lib' + '$ORIGIN/../../nvidia/cudnn/lib' + '$ORIGIN/../../nvidia/cufft/lib' + '$ORIGIN/../../nvidia/curand/lib' + '$ORIGIN/../../nvidia/cusolver/lib' + '$ORIGIN/../../nvidia/cusparse/lib' + '$ORIGIN/../../nvidia/nccl/lib' + '$ORIGIN/../../nvidia/nvtx/lib' + ) + CUDA_RPATHS=$(IFS=: ; echo "${CUDA_RPATHS[*]}") + export C_SO_RPATH=$CUDA_RPATHS':$ORIGIN:$ORIGIN/lib' + export LIB_SO_RPATH=$CUDA_RPATHS':$ORIGIN' + export FORCE_RPATH="--force-rpath" + export USE_STATIC_NCCL=0 + export USE_SYSTEM_NCCL=1 + export ATEN_STATIC_CUDA=0 + export USE_CUDA_STATIC_LINK=0 + export USE_CUPTI_SO=1 + export NCCL_INCLUDE_DIR="/usr/local/cuda/include/" + export NCCL_LIB_DIR="/usr/local/cuda/lib64/" + fi else echo "Unknown cuda version $CUDA_VERSION" exit 1 fi +# TODO: Remove me when Triton has a proper release channel +if [[ $(uname) == "Linux" && -z "$PYTORCH_EXTRA_INSTALL_REQUIREMENTS" ]]; then + TRITON_SHORTHASH=$(cut -c1-10 $PYTORCH_ROOT/.github/ci_commit_pins/triton.txt) + export PYTORCH_EXTRA_INSTALL_REQUIREMENTS="pytorch-triton==2.1.0+${TRITON_SHORTHASH}" +fi + # builder/test.sh requires DESIRED_CUDA to know what tests to exclude export DESIRED_CUDA="$cuda_version_nodot" diff --git a/manywheel/build_docker.sh b/manywheel/build_docker.sh index 1b8b04e706..9b0480210f 100755 --- a/manywheel/build_docker.sh +++ b/manywheel/build_docker.sh @@ -56,7 +56,7 @@ case ${GPU_ARCH_TYPE} in if [[ $ROCM_VERSION_INT -ge 40300 ]]; then PYTORCH_ROCM_ARCH="${PYTORCH_ROCM_ARCH};gfx90a;gfx1030" fi - DOCKER_GPU_BUILD_ARG="--build-arg ROCM_VERSION=${GPU_ARCH_VERSION} --build-arg PYTORCH_ROCM_ARCH=${PYTORCH_ROCM_ARCH}" + DOCKER_GPU_BUILD_ARG="--build-arg ROCM_VERSION=${GPU_ARCH_VERSION} --build-arg PYTORCH_ROCM_ARCH=${PYTORCH_ROCM_ARCH} --build-arg DEVTOOLSET_VERSION=9" ;; *) echo "ERROR: Unrecognized GPU_ARCH_TYPE: ${GPU_ARCH_TYPE}" diff --git a/manywheel/build_libtorch.sh b/manywheel/build_libtorch.sh index 855d4bcc83..32e0f7a1a1 100644 --- a/manywheel/build_libtorch.sh +++ b/manywheel/build_libtorch.sh @@ -65,13 +65,11 @@ fi # ever pass one python version, so we assume that DESIRED_PYTHON is not a list # in this case if [[ -n "$DESIRED_PYTHON" && "$DESIRED_PYTHON" != cp* ]]; then - if [[ "$DESIRED_PYTHON" == '2.7mu' ]]; then - DESIRED_PYTHON='cp27-cp27mu' - elif [[ "$DESIRED_PYTHON" == '3.8m' ]]; then - DESIRED_PYTHON='cp38-cp38' + if [[ "$DESIRED_PYTHON" == '3.7' ]]; then + DESIRED_PYTHON='cp37-cp37m' else python_nodot="$(echo $DESIRED_PYTHON | tr -d m.u)" - DESIRED_PYTHON="cp${python_nodot}-cp${python_nodot}m" + DESIRED_PYTHON="cp${python_nodot}-cp${python_nodot}" fi fi pydir="/opt/python/$DESIRED_PYTHON" diff --git a/release/promote.sh b/release/promote.sh index 984788e42b..1147dc0c98 100644 --- a/release/promote.sh +++ b/release/promote.sh @@ -6,10 +6,11 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" source "${DIR}/release_versions.sh" # Make sure to update these versions when doing a release first -PYTORCH_VERSION=${PYTORCH_VERSION:-1.12.0} -TORCHVISION_VERSION=${TORCHVISION_VERSION:-0.13.0} -TORCHAUDIO_VERSION=${TORCHAUDIO_VERSION:-0.12.0} -TORCHTEXT_VERSION=${TORCHTEXT_VERSION:-0.13.0} +PYTORCH_VERSION=${PYTORCH_VERSION:-2.0.0} +TORCHVISION_VERSION=${TORCHVISION_VERSION:-0.15.0} +TORCHAUDIO_VERSION=${TORCHAUDIO_VERSION:-2.0.0} +TORCHTEXT_VERSION=${TORCHTEXT_VERSION:-0.15.0} +TORCHDATA_VERSION=${TORCHDATA_VERSION:-0.6.0} DRY_RUN=${DRY_RUN:-enabled} @@ -70,16 +71,48 @@ promote_pypi() { echo } +# Promote s3 dependencies +# promote_s3 "certifi" whl "2022.12.7" +# promote_s3 "charset_normalizer" whl "2.1.1" +# promote_s3 "cmake" whl "3.25" +# promote_s3 "colorama" whl "0.4.6" +# promote_s3 "triton" whl "2.0.0" +# promote_s3 "pytorch_triton_rocm" whl "2.0.1" +# promote_s3 "tqdm" whl "4.64.1" +# promote_s3 "Pillow" whl "9.3.0" +# for python 3.8-3.11 +# promote_s3 "numpy" whl "1.24.1" +# for python 3.7 older pytorch versions +# promote_s3 "numpy" whl "1.21.6" +# promote_s3 "urllib3" whl "1.26.13" +# promote_s3 "lit" whl "15.0.7" +# promote_s3 "sympy" whl "1.11.1" +# promote_s3 "typing_extensions" whl "4.4.0" +# promote_s3 "filelock" whl "3.9.0" +# promote_s3 "mpmath" whl "1.2.1" +# promote_s3 "MarkupSafe" whl "2.1.2" +# promote_s3 "Jinja2" whl "3.1.2" +# promote_s3 "idna" whl "3.4" +# promote_s3 "networkx" whl "3.0" +# promote_s3 "packaging" whl "22.0" +# promote_s3 "requests" whl "2.28.1" + # promote_s3 torch whl "${PYTORCH_VERSION}" # promote_s3 torchvision whl "${TORCHVISION_VERSION}" # promote_s3 torchaudio whl "${TORCHAUDIO_VERSION}" # promote_s3 torchtext whl "${TORCHTEXT_VERSION}" +# promote_s3 torchdata whl "${TORCHDATA_VERSION}" # promote_s3 "libtorch-*" libtorch "${PYTORCH_VERSION}" +# promote_conda torchtriton conda "2.0.0" +# promote_conda pytorch-cuda conda "11.7" +# promote_conda pytorch-cuda conda "11.8" + # promote_conda pytorch conda "${PYTORCH_VERSION}" # promote_conda torchvision conda "${TORCHVISION_VERSION}" # promote_conda torchaudio conda "${TORCHAUDIO_VERSION}" # promote_conda torchtext conda "${TORCHTEXT_VERSION}" +# promote_conda torchdata conda "${TORCHDATA_VERSION}" # Uncomment these to promote to pypi LINUX_VERSION_SUFFIX="%2Bcu102" diff --git a/release/pypi/prep_binary_for_pypi.sh b/release/pypi/prep_binary_for_pypi.sh old mode 100644 new mode 100755 index 201e4b9ac5..fdd9bf4a0e --- a/release/pypi/prep_binary_for_pypi.sh +++ b/release/pypi/prep_binary_for_pypi.sh @@ -12,6 +12,19 @@ set -eou pipefail shopt -s globstar +# Function copied from manywheel/build_common.sh +make_wheel_record() { + FPATH=$1 + if echo $FPATH | grep RECORD >/dev/null 2>&1; then + # if the RECORD file, then + echo "$FPATH,," + else + HASH=$(openssl dgst -sha256 -binary $FPATH | openssl base64 | sed -e 's/+/-/g' | sed -e 's/\//_/g' | sed -e 's/=//g') + FSIZE=$(ls -nl $FPATH | awk '{print $5}') + echo "$FPATH,sha256=$HASH,$FSIZE" + fi +} + OUTPUT_DIR=${OUTPUT_DIR:-$(pwd)} tmp_dir="$(mktemp -d)" @@ -27,8 +40,9 @@ for whl_file in "$@"; do set -x unzip -q "${whl_file}" -d "${whl_dir}" ) - version_with_suffix=$(grep '^Version:' "${whl_dir}"/*/METADATA | cut -d' ' -f2) + version_with_suffix=$(grep '^Version:' "${whl_dir}"/*/METADATA | cut -d' ' -f2 | tr -d "[:space:]") version_with_suffix_escaped=${version_with_suffix/+/%2B} + # Remove all suffixed +bleh versions version_no_suffix=${version_with_suffix/+*/} new_whl_file=${OUTPUT_DIR}/$(basename "${whl_file/${version_with_suffix_escaped}/${version_no_suffix}}") @@ -37,11 +51,37 @@ for whl_file in "$@"; do dirname_dist_info_folder=$(dirname "${dist_info_folder}") ( set -x + + # Special build with pypi cudnn remove it from version + if [[ $whl_file == *"with.pypi.cudnn"* ]]; then + rm -rf "${whl_dir}/caffe2" + rm -rf "${whl_dir}"/torch/lib/libnvrtc* + + sed -i -e "s/-with-pypi-cudnn//g" "${whl_dir}/torch/version.py" + fi + find "${dist_info_folder}" -type f -exec sed -i "s!${version_with_suffix}!${version_no_suffix}!" {} \; # Moves distinfo from one with a version suffix to one without # Example: torch-1.8.0+cpu.dist-info => torch-1.8.0.dist-info mv "${dist_info_folder}" "${dirname_dist_info_folder}/${basename_dist_info_folder/${version_with_suffix}/${version_no_suffix}}" cd "${whl_dir}" - zip -qr "${new_whl_file}" . + + ( + set +x + # copied from manywheel/build_common.sh + # regenerate the RECORD file with new hashes + record_file="${dirname_dist_info_folder}/${basename_dist_info_folder/${version_with_suffix}/${version_no_suffix}}/RECORD" + if [[ -e $record_file ]]; then + echo "Generating new record file $record_file" + : > "$record_file" + # generate records for folders in wheel + find * -type f | while read fname; do + make_wheel_record "$fname" >>"$record_file" + done + fi + ) + + rm -rf "${new_whl_file}" + zip -qr9 "${new_whl_file}" . ) done diff --git a/release/pypi/promote_pypi_to_staging.sh b/release/pypi/promote_pypi_to_staging.sh index 02ebe4833c..74f139680e 100644 --- a/release/pypi/promote_pypi_to_staging.sh +++ b/release/pypi/promote_pypi_to_staging.sh @@ -21,16 +21,17 @@ upload_pypi_to_staging() { } # Uncomment these to promote to pypi -LINUX_VERSION_SUFFIX="%2Bcu102" +PYTORCH_LINUX_VERSION_SUFFIX="%2Bcu117.with.pypi.cudnn" +LINUX_VERSION_SUFFIX="%2Bcu117" WIN_VERSION_SUFFIX="%2Bcpu" MACOS_X86_64="macosx_.*_x86_64" MACOS_ARM64="macosx_.*_arm64" -PLATFORM="linux_x86_64" VERSION_SUFFIX="${LINUX_VERSION_SUFFIX}" upload_pypi_to_staging torch "${PYTORCH_VERSION}" -PLATFORM="manylinux2014_aarch64" VERSION_SUFFIX="" upload_pypi_to_staging torch "${PYTORCH_VERSION}" -PLATFORM="win_amd64" VERSION_SUFFIX="${WIN_VERSION_SUFFIX}" upload_pypi_to_staging torch "${PYTORCH_VERSION}" -PLATFORM="${MACOS_X86_64}" VERSION_SUFFIX="" upload_pypi_to_staging torch "${PYTORCH_VERSION}" # intel mac -PLATFORM="${MACOS_ARM64}" VERSION_SUFFIX="" upload_pypi_to_staging torch "${PYTORCH_VERSION}" # m1 mac +PLATFORM="linux_x86_64" VERSION_SUFFIX="${PYTORCH_LINUX_VERSION_SUFFIX}" upload_pypi_to_staging torch "${PYTORCH_VERSION}" +PLATFORM="manylinux2014_aarch64" VERSION_SUFFIX="" upload_pypi_to_staging torch "${PYTORCH_VERSION}" +PLATFORM="win_amd64" VERSION_SUFFIX="${WIN_VERSION_SUFFIX}" upload_pypi_to_staging torch "${PYTORCH_VERSION}" +PLATFORM="${MACOS_X86_64}" VERSION_SUFFIX="" upload_pypi_to_staging torch "${PYTORCH_VERSION}" # intel mac +PLATFORM="${MACOS_ARM64}" VERSION_SUFFIX="" upload_pypi_to_staging torch "${PYTORCH_VERSION}" # m1 mac PLATFORM="linux_x86_64" VERSION_SUFFIX="${LINUX_VERSION_SUFFIX}" upload_pypi_to_staging torchvision "${TORCHVISION_VERSION}" PLATFORM="manylinux2014_aarch64" VERSION_SUFFIX="" upload_pypi_to_staging torchvision "${TORCHVISION_VERSION}" diff --git a/release/release_versions.sh b/release/release_versions.sh index 95ebfa363b..f0db2a0895 100644 --- a/release/release_versions.sh +++ b/release/release_versions.sh @@ -1,7 +1,8 @@ #!/usr/bin/env bash # Make sure to update these versions when doing a release first -PYTORCH_VERSION=${PYTORCH_VERSION:-1.12.0} -TORCHVISION_VERSION=${TORCHVISION_VERSION:-0.13.0} -TORCHAUDIO_VERSION=${TORCHAUDIO_VERSION:-0.12.0} -TORCHTEXT_VERSION=${TORCHTEXT_VERSION:-0.13.0} +PYTORCH_VERSION=${PYTORCH_VERSION:-2.0.0} +TORCHVISION_VERSION=${TORCHVISION_VERSION:-0.15.0} +TORCHAUDIO_VERSION=${TORCHAUDIO_VERSION:-2.0.0} +TORCHTEXT_VERSION=${TORCHTEXT_VERSION:-0.15.0} +TORCHDATA_VERSION=${TORCHDATA_VERSION:-0.6.0} diff --git a/run_tests.sh b/run_tests.sh index 18b00f00b3..fd66835e23 100755 --- a/run_tests.sh +++ b/run_tests.sh @@ -72,21 +72,6 @@ fi # Environment initialization if [[ "$package_type" == conda || "$(uname)" == Darwin ]]; then - # Why are there two different ways to install dependencies after installing an offline package? - # The "cpu" conda package for pytorch doesn't actually depend on "cpuonly" which means that - # when we attempt to update dependencies using "conda update --all" it will attempt to install - # whatever "cudatoolkit" your current computer relies on (which is sometimes none). When conda - # tries to install this cudatoolkit that correlates with your current hardware it will also - # overwrite the currently installed "local" pytorch package meaning you aren't actually testing - # the right package. - # TODO (maybe): Make the "cpu" package of pytorch depend on "cpuonly" - if [[ "$cuda_ver" = 'cpu' ]]; then - # Installing cpuonly will also install dependencies as well - retry conda install -y -c pytorch cpuonly - else - # Install dependencies from installing the pytorch conda package offline - retry conda update -yq --all -c defaults -c pytorch -c numba/label/dev - fi # Install the testing dependencies retry conda install -yq future hypothesis ${NUMPY_PACKAGE} ${PROTOBUF_PACKAGE} pytest setuptools six typing_extensions pyyaml else @@ -140,15 +125,21 @@ python -c "import torch; exit(0 if torch.__version__ == '$expected_version' else # Test that CUDA builds are setup correctly if [[ "$cuda_ver" != 'cpu' ]]; then - # Test CUDA archs - echo "Checking that CUDA archs are setup correctly" - timeout 20 python -c 'import torch; torch.randn([3,5]).cuda()' - - # These have to run after CUDA is initialized - echo "Checking that magma is available" - python -c 'import torch; torch.rand(1).cuda(); exit(0 if torch.cuda.has_magma else 1)' - echo "Checking that CuDNN is available" - python -c 'import torch; exit(0 if torch.backends.cudnn.is_available() else 1)' + cuda_installed=1 + nvidia-smi || cuda_installed=0 + if [[ "$cuda_installed" == 0 ]]; then + echo "Skip CUDA tests for machines without a Nvidia GPU card" + else + # Test CUDA archs + echo "Checking that CUDA archs are setup correctly" + timeout 20 python -c 'import torch; torch.randn([3,5]).cuda()' + + # These have to run after CUDA is initialized + echo "Checking that magma is available" + python -c 'import torch; torch.rand(1).cuda(); exit(0 if torch.cuda.has_magma else 1)' + echo "Checking that CuDNN is available" + python -c 'import torch; exit(0 if torch.backends.cudnn.is_available() else 1)' + fi fi # Check that OpenBlas is not linked to on Macs diff --git a/s3_management/backup_conda.py b/s3_management/backup_conda.py index a75c23407d..06926589d3 100644 --- a/s3_management/backup_conda.py +++ b/s3_management/backup_conda.py @@ -1,7 +1,8 @@ #!/usr/bin/env python3 -# Downloads domain library packages from channel +# Downloads domain pytorch and library packages from channel # And backs them up to S3 # Do not use unless you know what you are doing +# Usage: python backup_conda.py --version 1.6.0 import conda.api import boto3 @@ -9,6 +10,7 @@ import urllib import os import hashlib +import argparse S3 = boto3.resource('s3') BUCKET = S3.Bucket('pytorch-backup') @@ -23,11 +25,13 @@ def compute_md5(path:str) -> str: def download_conda_package(package:str, version:Optional[str] = None, depends:Optional[str] = None, channel:Optional[str] = None) -> List[str]: packages = conda.api.SubdirData.query_all(package, channels = [channel] if channel is not None else None, subdirs = _known_subdirs) rc = [] + for pkg in packages: if version is not None and pkg.version != version: continue if depends is not None and depends not in pkg.depends: continue + print(f"Downloading {pkg.url}...") os.makedirs(pkg.subdir, exist_ok = True) fname = f"{pkg.subdir}/{pkg.fn}" @@ -50,6 +54,18 @@ def upload_to_s3(prefix: str, fnames: List[str]) -> None: if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + "--version", + help="PyTorch Version to backup", + type=str, + required = True + ) + options = parser.parse_args() + rc = download_conda_package("pytorch", channel = "pytorch", version = options.version) + upload_to_s3(f"v{options.version}/conda", rc) + for libname in ["torchvision", "torchaudio", "torchtext"]: - rc = download_conda_package(libname, channel = "pytorch", depends = "pytorch 1.9.0") - upload_to_s3("v1.9.0-rc4/conda", rc) + print(f"processing {libname}") + rc = download_conda_package(libname, channel = "pytorch", depends = f"pytorch {options.version}") + upload_to_s3(f"v{options.version}/conda", rc) diff --git a/s3_management/manage.py b/s3_management/manage.py index 6b6d0c6faa..15b37cf3da 100644 --- a/s3_management/manage.py +++ b/s3_management/manage.py @@ -1,16 +1,15 @@ #!/usr/bin/env python import argparse -import tempfile import time from os import path, makedirs +from datetime import datetime from collections import defaultdict from typing import Iterator, List, Type, Dict, Set, TypeVar, Optional -from re import sub, match +from re import sub, match, search from packaging.version import parse -import botocore import boto3 @@ -18,7 +17,7 @@ CLIENT = boto3.client('s3') BUCKET = S3.Bucket('pytorch') -ACCEPTED_FILE_EXTENSIONS = ("whl", "zip") +ACCEPTED_FILE_EXTENSIONS = ("whl", "zip", "tar.gz") ACCEPTED_SUBDIR_PATTERNS = [ r"cu[0-9]+", # for cuda r"rocm[0-9]+\.[0-9]+", # for rocm @@ -31,11 +30,71 @@ "whl/test": "torch_test.html", } +# NOTE: This refers to the name on the wheels themselves and not the name of +# package as specified by setuptools, for packages with "-" (hyphens) in their +# names you need to convert them to "_" (underscores) in order for them to be +# allowed here since the name of the wheels is compared here +PACKAGE_ALLOW_LIST = { + "Pillow", + "certifi", + "charset_normalizer", + "cmake", + "colorama", + "filelock", + "idna", + "Jinja2", + "lit", + "MarkupSafe", + "mpmath", + "nestedtensor", + "networkx", + "numpy", + "packaging", + "portalocker", + "pytorch_triton", + "pytorch_triton_rocm", + "requests", + "sympy", + "torch", + "torcharrow", + "torchaudio", + "torchcsprng", + "torchdata", + "torchdistx", + "torchrec", + "torchtext", + "torchvision", + "triton", + "tqdm", + "typing_extensions", + "urllib3", +} + +# Should match torch-2.0.0.dev20221221+cu118-cp310-cp310-linux_x86_64.whl as: +# Group 1: torch-2.0.0.dev +# Group 2: 20221221 +PACKAGE_DATE_REGEX = r"([a-zA-z]*-[0-9.]*.dev)([0-9]*)" + # How many packages should we keep of a specific package? KEEP_THRESHOLD = 60 S3IndexType = TypeVar('S3IndexType', bound='S3Index') +def extract_package_build_time(full_package_name: str) -> datetime: + result = search(PACKAGE_DATE_REGEX, full_package_name) + if result is not None: + try: + return datetime.strptime(result.group(2), "%Y%m%d") + except ValueError: + # Ignore any value errors since they probably shouldn't be hidden anyways + pass + return datetime.now() + +def between_bad_dates(package_build_time: datetime): + start_bad = datetime(year=2022, month=8, day=17) + end_bad = datetime(year=2022, month=12, day=30) + return start_bad <= package_build_time <= end_bad + class S3Index: def __init__(self: S3IndexType, objects: List[str], prefix: str) -> None: @@ -70,9 +129,17 @@ def nightly_packages_to_show(self: S3IndexType) -> Set[str]: packages: Dict[str, int] = defaultdict(int) to_hide: Set[str] = set() for obj in all_sorted_packages: - package_name = path.basename(obj).split('-')[0] + full_package_name = path.basename(obj) + package_name = full_package_name.split('-')[0] + package_build_time = extract_package_build_time(full_package_name) + # Hard pass on packages that are included in our allow list + if package_name not in PACKAGE_ALLOW_LIST: + to_hide.add(obj) + continue if packages[package_name] >= KEEP_THRESHOLD: to_hide.add(obj) + elif between_bad_dates(package_build_time): + to_hide.add(obj) else: packages[package_name] += 1 return set(self.objects).difference({ @@ -162,7 +229,7 @@ def to_simple_package_html( out.append('') out.append('') out.append(' ') - out.append('

Links for {}

'.format(package_name)) + out.append('

Links for {}

'.format(package_name.lower().replace("_","-"))) for obj in sorted(self.gen_file_list(subdir, package_name)): out.append(f' {path.basename(obj).replace("%2B","+")}
') # Adding html footer @@ -183,7 +250,7 @@ def to_simple_packages_html( out.append('') out.append(' ') for pkg_name in sorted(self.get_package_names(subdir)): - out.append(f' {pkg_name}
') + out.append(f' {pkg_name.replace("_","-")}
') # Adding html footer out.append(' ') out.append('') @@ -214,9 +281,10 @@ def upload_pep503_htmls(self) -> None: Body=self.to_simple_packages_html(subdir=subdir) ) for pkg_name in self.get_package_names(subdir=subdir): - print(f"INFO Uploading {subdir}/{pkg_name}/index.html") + compat_pkg_name = pkg_name.lower().replace("_", "-") + print(f"INFO Uploading {subdir}/{compat_pkg_name}/index.html") BUCKET.Object( - key=f"{subdir}/{pkg_name}/index.html" + key=f"{subdir}/{compat_pkg_name}/index.html" ).put( ACL='public-read', CacheControl='no-cache,no-store,must-revalidate', diff --git a/s3_management/requirements.txt b/s3_management/requirements.txt index 86199dbc6e..d9fe7f1f00 100644 --- a/s3_management/requirements.txt +++ b/s3_management/requirements.txt @@ -1,2 +1,2 @@ -boto3 -packaging +boto3==1.12.7 +packaging==21.3 diff --git a/smoke_test.sh b/smoke_test.sh deleted file mode 100755 index e2459b49d6..0000000000 --- a/smoke_test.sh +++ /dev/null @@ -1,197 +0,0 @@ -#!/bin/bash -set -eux -o pipefail -SOURCE_DIR=$(cd $(dirname $0) && pwd) - -# This is meant to be run in either a docker image or in a Mac. This assumes an -# environment that will be teared down after execution is finishes, so it will -# probably mess up what environment it runs in. - -# This is now only meant to be run in CircleCI, after calling the -# .circleci/scripts/binary_populate_env.sh . You can call this manually if you -# make sure all the needed variables are still populated. - -# Function to retry functions that sometimes timeout or have flaky failures -retry () { - $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) -} - -if ! [ -x "$(command -v curl)" ]; then - if [ -f /etc/lsb-release ]; then - # TODO: Remove this once nvidia package repos are back online - # Comment out nvidia repositories to prevent them from getting apt-get updated, see https://github.com/pytorch/pytorch/issues/74968 - # shellcheck disable=SC2046 - sed -i 's/.*nvidia.*/# &/' $(find /etc/apt/ -type f -name "*.list") - - apt-get update - apt-get install -y curl - fi -fi - -# Use today's date if none is given -if [[ -z "${DATE:-}" || "${DATE:-}" == 'today' ]]; then - DATE="$(date +%Y%m%d)" -fi - -# DESIRED_PYTHON is in format 2.7m?u? -# DESIRED_CUDA is in format cu80 (or 'cpu') - -if [[ "$DESIRED_CUDA" == cpu ]]; then - export USE_CUDA=0 -else - export USE_CUDA=1 -fi - -# Generate M.m formats for CUDA and Python versions -if [[ "$DESIRED_CUDA" != cpu ]]; then - cuda_dot="$(echo $DESIRED_CUDA | tr -d 'cpu')" - if [[ "${#cuda_dot}" == 2 ]]; then - cuda_dot="${cuda_dot:0:1}.${cuda_dot:1}" - else - cuda_dot="${cuda_dot:0:2}.${cuda_dot:2}" - fi -fi -py_dot="${DESIRED_PYTHON:0:3}" - -# Generate "long" python versions cp27-cp27mu -py_long="cp${DESIRED_PYTHON:0:1}${DESIRED_PYTHON:2:1}-cp${DESIRED_PYTHON:0:1}${DESIRED_PYTHON:2}" -# TODO: I know this is the wrong way to do this translation, we should probably fix it upstream, but this is the quickest way -if [[ "${py_long}" = "cp38-cp38m" ]]; then - py_long="cp38-cp38" -fi - -# Determine package name -if [[ "$PACKAGE_TYPE" == 'libtorch' ]]; then - if [[ "$(uname)" == Darwin ]]; then - libtorch_variant='macos' - elif [[ -z "${LIBTORCH_VARIANT:-}" ]]; then - echo "No libtorch variant given. This smoke test does not know which zip" - echo "to download." - exit 1 - else - libtorch_variant="$LIBTORCH_VARIANT" - fi - if [[ "$DESIRED_DEVTOOLSET" == *"cxx11-abi"* ]]; then - LIBTORCH_ABI="cxx11-abi-" - else - LIBTORCH_ABI= - fi - if [[ "$DESIRED_CUDA" == 'cu102' || "$libtorch_variant" == 'macos' ]]; then - package_name="libtorch-$LIBTORCH_ABI$libtorch_variant-${NIGHTLIES_DATE_PREAMBLE}${DATE}.zip" - else - package_name="libtorch-$LIBTORCH_ABI$libtorch_variant-${NIGHTLIES_DATE_PREAMBLE}${DATE}%2B${DESIRED_CUDA}.zip" - fi - -elif [[ "$PACKAGE_TYPE" == *wheel ]]; then - package_name='torch' -else - package_name='pytorch' -fi -if [[ "$(uname)" == 'Darwin' ]] || [[ "$DESIRED_CUDA" == "cu102" ]] || [[ "$PACKAGE_TYPE" == 'conda' ]]; then - package_name_and_version="${package_name}==${NIGHTLIES_DATE_PREAMBLE}${DATE}" -else - # Linux binaries have the cuda version appended to them. This is only on - # linux, since all macos builds are cpu. (NB: We also omit - # DESIRED_CUDA if it's the default) - package_name_and_version="${package_name}==${NIGHTLIES_DATE_PREAMBLE}${DATE}+${DESIRED_CUDA}" -fi - -# Switch to the desired python -if [[ "$PACKAGE_TYPE" == 'conda' || "$(uname)" == 'Darwin' ]]; then - # Create a new conda env in conda, or on MacOS - conda create -yn test python="$py_dot" && source activate test - python_version=$(python --version 2>&1) - dependencies="numpy protobuf six requests" - case ${python_version} in - *3.6.*) - dependencies="${dependencies} future dataclasses" - ;; - esac - conda install -yq ${dependencies} -else - export PATH=/opt/python/${py_long}/bin:$PATH - if [[ "$(python --version 2>&1)" == *3.6.* ]]; then - retry pip install -q future numpy protobuf six requests dataclasses - else - retry pip install -q future numpy protobuf six requests - fi -fi - -# Switch to the desired CUDA if using the conda-cuda Docker image -if [[ "$PACKAGE_TYPE" == 'conda' ]]; then - rm -rf /usr/local/cuda || true - if [[ "$DESIRED_CUDA" != 'cpu' ]]; then - ln -s "/usr/local/cuda-${cuda_dot}" /usr/local/cuda - export CUDA_VERSION=$(ls /usr/local/cuda/lib64/libcudart.so.*|sort|tac | head -1 | rev | cut -d"." -f -3 | rev) # 10.0.130 - export CUDA_VERSION_SHORT=$(ls /usr/local/cuda/lib64/libcudart.so.*|sort|tac | head -1 | rev | cut -d"." -f -3 | rev | cut -f1,2 -d".") # 10.0 - export CUDNN_VERSION=$(ls /usr/local/cuda/lib64/libcudnn.so.*|sort|tac | head -1 | rev | cut -d"." -f -3 | rev) - fi -fi - -# Print some debugging info -python --version -pip --version -which python -# If you are debugging packages not found then run these commands. -#if [[ "$PACKAGE_TYPE" == 'conda' ]]; then -# conda search -c pytorch "$package_name" -#elif [[ "$PACKAGE_TYPE" == *wheel ]]; then -# retry curl "https://download.pytorch.org/whl/nightly/$DESIRED_CUDA/torch_nightly.html" -v -#fi - -# Install the package for the requested date -if [[ "$PACKAGE_TYPE" == 'libtorch' ]]; then - mkdir tmp_libtorch - pushd tmp_libtorch - libtorch_url="https://download.pytorch.org/libtorch/nightly/$DESIRED_CUDA/$package_name" - retry curl -o libtorch_zip "${libtorch_url}" - unzip -q libtorch_zip - cd libtorch -elif [[ "$PACKAGE_TYPE" == 'conda' ]]; then - if [[ "$DESIRED_CUDA" == 'cpu' ]]; then - if [[ "$(uname)" == 'Darwin' ]]; then - retry conda install -yq -c pytorch-nightly "$package_name_and_version" - else - retry conda install -yq -c pytorch-nightly "$package_name_and_version" cpuonly - fi - else - retry conda install -yq -c pytorch-nightly "cudatoolkit=$CUDA_VERSION_SHORT" "$package_name_and_version" - fi -else - # We need to upgrade pip now that we have '+cuver' in the package name, as - # old pips do not correctly change the '+' to '%2B' in the url and fail to - # find the package. - pip install --upgrade pip -q - pip_url="https://download.pytorch.org/whl/nightly/$DESIRED_CUDA/torch_nightly.html" - retry pip install "$package_name_and_version" \ - -f "$pip_url" \ - --no-cache-dir \ - --no-index \ - -q -fi - -# Check that all conda features are working -if [[ "$PACKAGE_TYPE" == 'conda' ]]; then - # Check that conda didn't change the Python version out from under us. Conda - # will do this if it didn't find the requested package for the current Python - # version and if nothing else has been installed in the current env. - if [[ -z "$(python --version 2>&1 | grep -o $py_dot)" ]]; then - echo "The Python version has changed to $(python --version)" - echo "Probably the package for the version we want does not exist" - echo '(conda will change the Python version even if it was explicitly declared)' - exit 1 - fi - - # Check that the CUDA feature is working - if [[ "$DESIRED_CUDA" == 'cpu' ]]; then - if [[ -n "$(conda list torch | grep -o cuda)" ]]; then - echo "The installed package is built for CUDA:: $(conda list torch)" - exit 1 - fi - elif [[ -z "$(conda list torch | grep -o cuda$cuda_dot)" ]]; then - echo "The installed package doesn't seem to be built for CUDA $cuda_dot" - echo "The full package is $(conda list torch)" - exit 1 - fi -fi - -"${SOURCE_DIR}/check_binary.sh" diff --git a/test/smoke_test/assets/dog2.jpg b/test/smoke_test/assets/dog2.jpg new file mode 100644 index 0000000000000000000000000000000000000000..528dfec7209cd3bbb415cf610a57b8955149d028 GIT binary patch literal 90796 zcmeFZbzEEB@;4gXY0;v^-9nHOq%H35))EpVK#)LjTBJpSYjJmXDbOOt-JRl6q%G2x z8aMRxInTN8@0|DE_x^j&3j4EXul1c-GqYyz?7hj%_2Tsg;I^8QsuBPL0|TIf{s68w zgR00mw1102pZMW~0Nv`MtMB)AWC62QGg)uMzB3;bTN8EPxt%yuf4LWrVcYmP#7o(0--s9AYnJ>)nE#MN zj>cpDp|gSL0d&Lv{ki^Ef8D5#MqaM~xqM2-J1pe(=afxuyJtl@CgWs(151f z0JNvDFtM?4aImq_=m7LM0GkYloJCj`mqHhU$LdB23X0FeXOpYzrqY}G$Sz`y3??9? zzC%Mx$HB?P&BH4yCVo!>EGe&`sHCi-s-|yXXk=_+Y6gYb*xK2{9o#*hczSvJ_=Y@x z5gHaAfl5eBN=|v1nwFklP*_x4Qd(AC-_Y39+|t_is;BpDU;n`1(D3xk?A-jq;?nZw z*7nZs-u}Vi`%h=*7nh&Ee7*X1W7mzHf3<%Y_P^LghPDe68ygE7@5U|+OfU3?MTU*T zB8*EetBVJ5qhJLE;Zw@R=hbx+u!-n>q_Rd%5mK{@ZgPCOG3~cy|L+V7{{LjzKMnia zu4Mo*76y9pu*d)p06l{0FYs!paI08(E8)S)`bIS@XG$0&&`e{nyNFLKLwZuByM{@K zbOBX95f$0Mm!IY0A;+}^&G%{#4|N1a<1E?Z&(>3qw3g#1Vf}`w%8ehY2&mYCbQoTP ztBAJZmLi6^vu0twe#ZCNJ?VouA`Z>^O!}MOeo`3$@~XHvECFklLY!B3U^9BdQUTyc zif9kUulgCn;zRE+%!7<27)PsR zmR6|q)r&WHBx)`{Z+pkK%7^$bX2zR~j&r7()G61%Iz}wz1{F41i8TF!67&Ote*^#D!`u*aoAHC@*z1qoB ziO+`p41dZh=u%wii)Pf8s}s{@+aWOeECLVAVZ@~1sw1g~;dfd~Ions2?$DH3weMh= zg73BYM@SXualh6hVcP_{bscF3Kk=X~>Ikot>K=!iOi3NU*18#2POz%1R+pQe|1xOh zUgIlzu8jXqc)z8|_ULo4H_ltTSheXj)fs43#|#k*FnZ}Ce|+BY2@x0GbYbvS)hy40 zEK@;StIeP?ms$?%$^+|GIc49&s@Va$%x$(NcQ5;Ga*9l&zN(?@IoGFF!YEcAdk^ZD z^k)P+MoD@+*g&cfYqd9{;3{upPc2tR=TPEJGtYsE#O$`x4~Ubhsvm1@$!&M%iex$n zXybBHFYYGCdhT2UD#46p#|lvg`|-OYb@j4Yvziz=0=AeZo@YK8q=lD6x#@AH`6 z$!U*C))lF!4Sf%{TOD=FPA=JT3e?!f2?NYA30S*{ujYck&5rqh(ZJYLmeu8`8iUqZ zv%Lv^n%qz#p(Q*9DVMlwy^q;c8KdHmXmaZ4Y$fk&5T#J0BzajfKx^Mi)+b51x+ZBr z&sw!yDu-jWw!)P(4Z~VAR(Eqat<+ZAzf|{BX~A^mjZ2H93tswQF7D9Jy3%@487aXL zpKe4W+r`!C1m6%hmNt~Z&RQ}%3d8F(Z=`?f`P1Pz1vN~dmBssfR8O-IQ7Wp4P<@{J zsn4K3%xwZf5YvY5iM; zenq&Fp2};_E=E4lM{g5s6}OvIU=)}Sd$NK+*sxpPs6r(f-c@_Fiz z;o|bC%~P2pb^`gt5Jay(y@i%lflE*-2{cM`nAQCK`CnAl&pp}Lc5tiA%_7`cnbu-{ zU1mb87p0Bj-;kLpBgk>E%F3Lk`4lWwgJ=N0bPEm5MzP)!R3R?3kPeSiK0ob5;o~uR z@rWX#hSJ#D3f|&-<06JuoArD;LaF7#I!;mFhYAH6U3sILk0hUd^v`cT6__(1=gr*S zS;nKB!Hp3zY4t87>pkn2;T=!h1K|NSo)nliy~`?)sCWf?a}Y^h4Fp#>AA}FA3#ud6 zl_99Xbfwm$HU`qm$<9K3vt7wAl^I23vAklKdG~kUutsjgG}+Ee$KYY+I(phjj4PVf znqu%Oo88S#lTrw)=9&Lo*PboG2sc|>`q=iRc+7n_Ifyc3byiX8J)E;DPhT$#k6^lF zDsos&eVF@59cjARrRNG39q~aaHYv$yn)}{PDG)Oz5gZ%`$du2gh7G#4Fr#V^gcfU5 z&#E0Q*-R)4?V@zDUvdnXGu+L)yHfU$X0L7OOJpOM8MnH|M4s`;Z{eP=_;{r-qdak_2?Of68&uGFc7@`X*tjMjcp-GYLc?S+u zIGc~$QI&zb+!f5(A~d(xc~-!XsL^H&B$ zUqJgo$2nNX(hY@j1VV8frh(FQaT`387M>E{2c^r1K+boQ`HhCygi_j(;H_7s0xZe( z!vtlv4KFZQ9u5>eI><_!=!65ZDr9n<5}_y__{qv(SIervTekawdf`0_o#GFQ8DoVODZF zFzPqm%?2FWpRXRQ8zaTXDCgFNM1>5AJ~}XJ09PTDvszKRmjt)30k_nSpA0t8xcKf? zpUf42^GH?NN;D~(c_r3)T88O}N$0ec9>%gJCN4Mb>F(CDHEAkW^$^L;XRsZ1-S(*lR~WgPK(-7EOIhO$af#|W#~;ayk9r%#5Xd}Q{_?UvtuE$(cEk$P z-4`xL4o}}Jli}quPT7!(VwewXd3zT^!&U^pQ#SJ8#mnPqDLTvLKA;M&0TJ+3?GzPak>!k+ zyd3)bp7=}Kh$ULl+_slqRqjS!6(t%NNu33804`3Eql@aiUM^jfS6fo<#t7;ZC}QT? zMTA|V2tV{@y7G-xrOH;Ct0{q2G<`d$BiXr!eNo1fp0mS}(Z!S4 z71R0mn#Tsx-(_nQRTchgkw&t55xxtFztX~nZGZ%8oA@ipLw2QhEwRu;CzhXAcrzLes0)W4p| z+$*DPErzcQ*pIx@6EPA~-J}V7`p0mVO|*d$z%+9gin< zVA1+IbX>NrLDTZ->=TE9H}p$EN2W(kckfGnDf5_8E08Q&I8mi>$ocL&6*{>sHX`p3 zp0<$=iGb#~eb8RYb8kQf*iOONyu^aG9KY^QXE+k8J=M8Y!ZbOpjj0Oc2Kmj4=WZW= zp25VA-ce>$xr$&gm_2pXRkd}7R=-PRhdO70G<7*J%GG!!_?gA6u6fI( zp5=3I*}zuXZO#-wsvJXAPE}T2YLIb-*H3thXANyc5Tk^^H;SUBumW z*CJ9izNc19M>-`@h4Q~j(+kR1uzaZqSfGm7UwVQ}_m*a)QmSN4`t`i*Ac$=)??q_^ z9;#7?J4){l*OV~8rJE5wcfG~zmtRE5XlxafGgywUgqh?EDyfvf7i_(0lDa5IO3ma( zeu$>%E7I6g`p*g26S+C>Anyf5hq1mQdC-4P6*CtrD5fbw99w~Gz8wA)ZgRIjoP9VY zd?wV3%~@*3dH1Y~3tWirRi=*;VrJ%nTH<1<;^s)QtP^`|Ncp_=O8BDIk@{2&L zVA;*0j_N5@v%DdRW1OZO5|qb5fl^OO(%77DnYmFm@^`ZxOZm=HSPan#vt`a~yS5QJ zbi5z$SqbhzjT&NfAS{UnI+LLeioqZP$KY3S=>K=@xaaSw7AxV=MiR624K79>zt*& zMV%Sh@cY+r8LdX|Y`)gd7D_SEF%KEs4NnC+8qZl@>;ZNGE{`rV9EM;d7m@BAp-Yz8o%+E8J zQu}2^y31gs*-Uca*6*m%&vm7`RVK|x-0p`Old7Pe%Nl6_S{YMiF@zG0dXSj0$MQ4j zni`v$8h)CbiHUMCa8h)~PfVPEXlE-WxSF4COOy&bFEHnRFrqaW8mjVX1nth+Cplj! ztEGh!9{Y!k^HS82;_}=MUc&oWZO54*Nv?V4p_DtyU${)FF5L+k-O5_>TyWap^P#!E ztq7g$xl-R&@@xaCq)t8P0UJuyjyWVoWX!?NDp~6z&`lQ!!;_KH-)V)1u zp6Wb%?wjsVu6iykDKLc0M;JOevAp9E?8VW6=A4hYeH+z$_SvT(+InKmFp=GtT)69p zqZ$#lrS90L1enr(&U|%KZF?K;#20fdZ>|S1s^ww4zIEDZ1B#Gew^~u(1F~lJJYt>c zXUS!MIxK`AJ6{Y{o>snt-4e9OJTV%O57Z*joxb>bBDB7>o3>-^)>7m#zq|@m_-Mg8 z~WUnPfF&KVyT{_WtxhG9L+nP-4fNXyMNo{!h zP{)C{yEo{v>ZslQjVGX-yj+wYf3My#IsI`ORo6TFN-0C4zEK$AWMh)L3%kSp6P;AM zVEcz~j#{J|s!T*5K)+7wAlCww z(HwhxPrkF~ru%er!I{&f3%lyVrypUI*{E$)dmU$EJqa^wU^$*s6l%)_5xq*ra62+6 z!SVGhXN=_o$7fzK&$8V0Ser9pL@%`V-?*c8pC6Ae5ao%En6R{r2wX(~!e%X!DG}Zn^=;9%Ox z6?L}sLzUZD)YAS=nirxQ^e1nX9UT>Z6|N3yh~40JCdWc z9M{@(6MV}sOIYvB>x()u?q!vjHv)tAier0)Rk;(4Fx;0j&CReZwWMI-fnY4rIbGWE zW32g3^2elyc($Jz4BQXtM1Me39;;1%7FpfQf9c2|F3ZT?h@(gy}VNLn&+{3^5pr5caEP6qT@o09hFgA!r6(*r^#xHZ$FzjpTqs} zGSq)pVL&=YQ*^bNqF#D8I9S43yGa|{c_)Z$`9b(c9EYR3ybXFo#f$yn66fq#TnAD{ z_?;5bUF8|&v${ubgkia5`R2pwz{0alq@4I-sAqZk?rQ;_tYB!)5>lKGOvs- zcXyjA)$7iAwwZran0QU)Wt}ORV1;Bqo4Bktnp(h_+~(k99f!hb2U+XHXl=81#E%m{ zq+duKse1)k<5!LhNg+in6hEg)zeKX@ZpJy&`vR*D=CqWalnyD&i!W&VEjkCE^Q+-o zT<+tYRe$f*h5PPBQm6SxP%Sg?r(V62!DAY4mAU_Q@p0h`@!=BUN(n1p)Q1amK1O+? z`u2u4*@(I2wH4AZ<~SMI>avRW?$egU56Hf1>rA;kCkv5w8J4^pc;qOh^>MGjKiGn| zucM6LXArAp`_ack7dMY-EnF)6ZbtXm^REj+RZYb&eW2K`yuI$62fIQ|h7F2J_w2QkT>Ld&V}-xGRkz+i04g*%?@bnas}Tazsj8Gl-WcBO(e`>>0x&)k(5# zQz{)EBl|!tAXh|#4f2AkO1CN0uM8A!5wiTMor!df-x{98BEwriM4^}O%b4U>wR2#; zUVM)OhArrBL@mlsddl<0Zsfybv%Pi^cJ)Am2)toQegDbtVY1mc#3goXV`5}E*u_&? z$GD=x?N*uJ1&@miW=ci7LDBcr0qOAp84HI%s2NuOd~&k3K+`mY+x zR9c=Ip+M`V5*U8zt?hMU0a0t}X5KB)U?PA|BGYi2dX~eEhQ{RZ5_a99)~!xomrzk} z^efXY4oke@?LbFA-6{TEYf#Sydy{^%`(wYFG7l;%&mdX-a6>qFunZku6(FAD)=`Dq!xetIqC}P3&F+=BtqlhAIm2>k55gfFSd4oQ0A6&GlBn zIqYfu2J^@evkax83y}`h&C*k6tUh{H?tbm3B01;trQVW5Cmm48K92p=BJR*pUF`$O z6X)uLLmMM2nn4|_FI~SZP=hTkCi^H?wLu06t~#nB$uqUlVGq`|@p08p!mBEbl7jG5Q^uty0;MXwn%*KC?e zD#cU2Of1OcJ;NdMU#J^=C%8v5^H%-^X@GrOZD~kJMS(j*?vWX&gm^@ zW?TJtC?lwz#s=F|bwP44W7oIY8Mtwx{0?t9Tfm;!%F-dL>gPNCcE%@~ynXRNniE`nSYdbpTX&30GSdTSZY48|N1__m8K$?%J>eaSKVeF2x;E16{+Zf^psEi@jO)KG>LytNz z9Vln4GKRAD29*Z&+Sb0#`Zj_z5L>iYIXSs_qH$N*PM(w8%t-Cpg4Ai@FPa{nR^p{m zq)SyiFuN8!m$4`#snx(#JZ91WMOAodSX3T!oWz(iXs=F-bU``YU*5u8S$JUZr}+8h zH)CSEU(Ac&=q|MHF0DOQWRVjm)vof%uk{XH7w-O2qmO?=boVLGBY2|p3S=d|vprVr zLyc2kO~{2$5|VT~{!7EV!$(e*3Ml-EvB5Flx!F$+*|&|HtQ4rlbYtB5@iwmr%RfML z%tEuFv&RXhu!mSSLEAyG=cuKGp%?A7ni*lkB3FdxS*&{y_P7Gj*j@2pw+WY#^y9|P zN9q+pjJ?nC_*%~%QVj@bHPI6=h7`EOK~%YBe>w_!r1tSDNL1f} zG2?$$j)l@5(BX1$a6 z>`TKJq(pVKqq0e%T`ZTIevGf@k7p^oiI8JGB^q{2@u*B8X&j z1d*?^L#3+)e3qK5Jco^bp4EZJDIFRM0MNNy4gB) z6Aj)GEmsEZelLmoB~f)m<iIfo`{k?R`UQfMZzJGLWo>;mie1s%&8=x6 ze}lo-2xey%ygdTU3tN!klcV4xNeFBzX$T14Q!le|!JXF+wE`*fv@3=;vQtEDr8Xb! zQ5&sfgK)s2HJL!}-kPGjJ(gI@5366rB^ZJLMY7DZDKB~WLBmB~YWr3CaX_cb!}An- zeF?Hy*c-+V+^X>*-z7^IeOsMkr|Xlq^On>I-Cijbe&K8U(OaY2&nNTY(fS5RMfrhm z-EMn|`U|To*@0pQqLFxK;{=0jd>KB*rMaQuXcvmdE>pX%vbH*H*8svZM;_%gc12%? z=uW=h1grg!rc7n>}zcKmT#xQ-US3icHjo>^deDTdp*8EMKE zKQB|>iAh@A`n?nPTaU?3Gk;VUt|6Ypk?t7o#pG*M^S0$~evoX%;85x|uE|d@<1@9f z84q{&!ootUMs|STy^HsJm;!3w6k8kPX%z}6?uO6|oG}%dMfOd1+k9pxrl7~--e)BE zQ6?(pA-cHMCmJWM*}_f?jsN)|S(fu9@QF?}2!#Hb#8SxH+iIVD8Zcs$5=5@90Xf<| z^ChCX%EXhk9A~($a*4X0X~kI@0fOQ!9igg=0F!EwcRxZbN%kt7hr6!65N?>5^BVz&);0~ z1zy(3nO+*8MfU=nH7Ri#lj8VN){?$qOip1$5IFgHAx5ZH}b zZ?t@8MJHZtoS5!{uPzlR!)p`$xYSBndX@x{?LS_!H%3xhJ1Ah4Hi){n7K474*>Qcv zFho>wIdtIgUS5)M1vAs&Id7?{jSbhKWlCM77w~drwy^=)$hRHR=ks&C$oFBgI@J#B zqR(TU#m&tZ8m#cr>M%z8qLRYy2`kO^L)r|7e3GT=RMGgTZBM)9uOWIUIV_5hsfzrM zpWpcT1?;rDXl=9Y{%~k~bbHP^qA-14HU?!v{Hv)V|M2W^c;TfgrLYjuUF2xMn5IK* zl(~&w5sxZ&%4I2M9L1ZFsO*wXH?ziqZL5PGT|2d@>sw;N>6OPAi2 zC2Ptuxl4c@yZcD!o@jLdHtQz6NJ^V2iZ58`b<%0Id*h4@QSX98 zow`Oc$gYH4=-IDMC;NJ3m$Eg*yJ~Ut%WW0v$VdkkPcNI^T4|hX0O^}#4{=o75JSQ9 zca3d~jxmKs&5~s!j#*@>8eTx1ika+^`Fa*Y-~jgTWH^*!Tzi84qrdcnrm5fe83%DSxyn7BK6ARwKk ztQ&BJ$uWM*J!jXlKfa`oKJo5JNmfbjk0UJQaC1TYSVxk)=8cPkXS|1q57d2ZLafna zK~X#D`xOFC2>f=<iXvT8kJwhz2(sl5|T-OtxrUbkx+?zF(EC>aqW;tV&!dQh49G zB*UD({%mEUIV&VBl6qj$ph(~|W?_uPRy><>7lt`rtgk0E%dpkGM;8le*h)-5T;hSn zndsN6JJ6Up9uI)i&}I3Uh{`b=^sHtqUUiYG&xXuB5yfFyfZRL}R)SZw%^n9 zDrTx7y#rVL;h6neF{YGCqKd3`x4@2&g^)h8GWJUOVBePovmEiZld#1BVY!k$=9reqQwk~Yk?mPJ@^F8UE;0L9Z#hL3h^7&EQs)(|; zOWwg7ImtRd;XGATEa(n0hkA6Z20) z?Pdl}e;p+bu$5g(Uw$*-1$K^Qb7|@(=Q&0P_^4?*n4EVJ3oUOb(}BD^XwWujgnFzk;wA*p716@DT)n1E}5qo5Xcmte-GUq$<4|p}H;YZg2{n zu3l8pAI$mwDdvstdW&#L^**DJ|=1hw*uFR2Ok(s;sl`G!1 zmVzgc*R&Ba)GEJ@j7G&LmBdpcbR@Gw&dQUVISknzYrbw|(W(7r*kZMKu)#p=9%M1P zpnJGmNM<9WQZetPH$+U1d1p-XD_gv&Wv{GSLQY3oty~$lD;|UAH~kf`W@ezAQ*ymh(lcV#8FU!ZKep> zg1R31>2*90X(FA}0!gCsvCX}^5yzt#S3YH|!T4-&j^v!ICFei`BGxO7yn9q~1QWFT zKX}SG7s#ajsc7&5m8EpcLN^KvBrC1 zlJ0TgWfxTQ3>wKGEMQ7>CRSEwZAW=|y`^QJlA;J^9@11zpfbx9d)X1mb3ou{oasUqZP)uq~|r@ej_uRKquv^fjFwPKrFo2 zBP~bufIiz;wm^oTKImKEEP-6z;fmW5Y}v}XS_U5R)@MU{1SH*k{d&Z5!KPm7nIIf8@s-fr-Nl#B>|LekPg0`m-?R@h|c~$SNc&d9k>c44Z7> z$4^I&YU!bE0s8WGV(=r33$b+{CVLh}aVJ`0&sui0qmSm`9+%nEmQ#GiA6my+*scK! z`n>RTx%LRsxL`Xa054+0C@_wGKcW}MyqypY+z{-Ke^fbzRwXqbM>H`~n}A&=H_aqMe`AP<8en(TAr!N12djVv1am^Wko5 zBr4}*L~sPlr>;%#ibR*3Q|VTAXtIyE$tK&?s1Z-!C`BYFfu@$1aUM6P5)#L-Y$FL{Yt^#Q}+E#%QY$f|gdU@0z*tDHzok9;@ zdVX=ObW<^nbJPvV`dpB+cpl)Md`w`Hl&KD7<~Z)#WV z(u>M^Xa6f}u%Ublug?U-(`4tVy)fOBEvu0|#psaJD01cPkZ)6<#za;An6AXLpcO{N zdwJO}Co7}s*xl|h3o{e{+@6e`d60EqDi&lg`dISkyA0`zUs;FK4^2KS55PB@H$t*= ze~Dh!hY$DJt!~`L6$v@|6>#{T?Rx2YleEa*!^2flNXXe;5CU_twh@H7AcVXju0q0s zKp}wi18-Ld)X~O++1kbq?kvOlq5TajGaM$vYAmJ+)O3}%v4^YpB5m}2wG5!Xj!-a+ z^}&5+X>Unyge$_v1H$Z$aB_B+^p;`$EnE^!-*gMHGXI8nILfe^pc6RcU63}+qJpA= zKml|ji6@8^EdvR&mDE#E`Xd2-B*XegRWC0uK`#+O7o?q#Fc=IL0)m7nd+U1Fd3@dv6--@9}po-q;jz6Mrj(-bw_kanh zq7OvThM)mLARtge00IyBqkp+6d{{ zxVt<-LTwa1ZJa&W{+i6y>52Ql&h^CI=BDenu1+vpA+(ymSL`qROLZpfZ=Odqh&f1^VQMT~$Gb z^=4K{mtBRHeisjsE7ni00x4I35Z%l#RVipLBawy))1gA7zDZpgNppcuZ@JG zLl)xnuln4`fuZHVM8w6!g~hD}#KmAh0Z|Avy&iD>Do#sf3mUL5Bq#0kN|Ya&Wb=`>nmCJkkc@;es@9adDDi{b!ET zO;G=dSV;{RDC8!pehT}A|0n7%em(TH#N7kwqxzrJ{=WhK5mhQ^4i}`)pSkpH-2PBbaOU5qmLvpv z;|>{CcZjDAjP*|^)E?q&XM?_6p~LteF!;ZTDxfe7C<1|q2}sz2U;^j}fC@<51EHf5 z43+?jf<>)?Fx$WQ-Cb-wydX#$Svz!yqt^x<;J@?Fn7RKE4?KSXz3gpn78{+BCIA!@ z5C$6j$w&k90ns^XK-S;ubaP4ihb=c}F#6>A+svEOTT=BVF7%(cx;nvakbm3$oA=)W z{_gw}LjRHJ@A2O}@-D7E=o7}?L)**w-%s^F5Wo2};7}W9cb9*w+`o(o-B|iB7h|+q zzq`;^_#5({Ul{k9|J(lSf&Y5ozaIFn2mb4U|9as6e;)Yzk<-Q*edFeZexSVG!mL%5 zmw&9Ir>&%_p@>e|N2lB8IzbWc*yv9_s|eSAaP{ibhZAAU=#2rO3{VAV0(1b(0BgV# z036^1@Bj#)_s-}ZcYq#xT<$;c)BWbxL33H7x!?dOnnMBL0zd#Dzxe?-Yk{6a)lrvzvsEB%z$ZY3jmyz001PW006}#06=K| z55J+uZpHxx3jlxt+E*H506;^E=q6B7~>6A=;-k&u$#CLtvwB_g^_eVdGel8TCo_|_d7YDyY%N-E0V zB_6QQJlHq{I5-59Bt#^X|HtjR=cdHN1;zyy2D-YzUnL$G&>s3-3*p8UzZ*6V7A__R z9snO*D1q)zp@hH2&|q}Y3v>a9>p#T?Ze!g>FK0D!1PAk&DnRV3Erm0RfyIU_@^v*} zI^<<`JXd#uFq7*KECIU{eC1x#$(>xH%%M;T#%`EHs zq~60_FobJbebqcE;hb(T!Jj>@*gA6tJhhn0ZF1W*8|DsGP0<%S5!hwepN^WRD^INu zCg%dH7|1ke3w1T&|5&T!@DsMN?{PS=>wy<92MtqmYe@>|*=z9Zi_K3bhff2qh+_(V8ibmt`O}>J$FXo~nBr$fC=uhHYehQco0gi%(3zp*V+rq-X9^xHQjK?zDDiWPm~Y7EZzh zSJ|9T1ydIg9G?zcn;dZvyD$nYWcY?ys@6etMnTf1##FsF;)-YdHr@rHuS2~xi+$;= zY|XgEqv)yrEW&V4U=~6&UDZrEhTnT`a9T%VCaw(BK!cw;c0}>6vc{`x~Ud)LFdcBU)Dcryfudf&xjOlEb~m%?1)JyviT* zA%y8uRL2FDnz$a~(k<+G4B>9Fg$$9-J`3>dN+~@FG}36@;)EqvP8XE1Y<%Wz5eX zWuS+=V_kKq1nW{oSIlM--V`E}F!mgJi#pLwtFA+@3W|+A7A1{>v3l`sWfwxO#!&|l zl?yvmKUY$sTT0@^O1Ve3vufybm=yjLqojWntS)WI{;fk_I&8RSTHe zVu0wfGO+#bR4U)D@Mg$mp-U{&!;0p&_$dx3ArNBGZX8iedAo>%p9N)1jQ2`1H&W5m zT22*wBCqwLZDP#ErYx6-KC34E2bLbh`oh1LkBIUcKa1*+PiXHb3Y<4(#9yynA=y=% z!bVm8qu2R~>q#8kR;*DzD4omRFcC2x?LOi^<}ejE`Y>tkZrpdOb)yvO5mgP2TsfS~ zUFr>O-PeHEm8KUDDJOR`jj1bolW)r`HLp9}F{!6Wsxb4ZVt?P!m*YZYj$BU8z}R=7 zX2XczPt{VxB0~xvX|}~VL#}=j$Tfe*RB5ZLHK;A8LFtIM-(L5OQt`@T*0{XC z`qO!vFHK~G>Ma0vI|){x)!m#oDS9wbl0^R2sjMuK)yTZzQH+t?FRb&){Ln;(uWC!< z&*V0xg_NbT*Ka8(S?Qw2j8=uw3h%DOA`U{Q0@EmLe~$Ir?z1=m3qqoYuUN|>7c zeVo1`MqCo&78mpa4xQdY!`-CK)~k9v3h~;m`!Y(D8uRR0n#plhn%UeY;at+?d`3@S zYpSIs^T-+Y;1mYtn#((8lmi`va68ej?VDNTs4YoT-xg&V7c#PE$LsA8@Zl6nl-`PL zjL^HVB63w$HBrr^;-Pq0p`p0NV@(lV>KO*RD(Q$~-jUQ`7~K~%4D>B*{6w2wM=!^o zsp*Dg4Yj^)5vekZEt8glYUZ*id|8|xyTPVpme!T zrMU8GA&Y3LZmV_+yVxG7%nVMXy4c#m;Tj!Uq}&^X zq85NQMzjxdFYN2ZF;3S-pwvc8tb-~;lJ0w+D%jTH4+Ilv;eavsTBmsOv*3oQPWq6f zo`tcgY+wG;C8;)nN(-g4APORtTcwG+iii7&6-f+Tj8TH4evfSMhW-8UiLg1xC{I8R zun~&-t*$ig*H~BZ<;MI1-0&DG0 zy|cd=-jdXWVd2H`?`5BL~VonA;#!ZM39XvywI zOpVIb@=VG^5{KcEBSOm%#!;9<@E4V|(pgx00{cgz@iR5p(SxEp%A|8EiU{`75|%(g zIik#c86R`?MpKV%0%jxvG|z5?yXS{#rWntIGlin|SeyHH4)#xE zJ!MOINwF*6zeEvBWs$+cN#Pn9*z1u^SH@*onWu^IBFXK?0s*B*Rt^`ZwHV)& zZC0)pw2a$xnA;EvodN29ak{ApYoSRK9^?aP6aEuYq>po3r#YWWHxj*L>{c3fTZ{jChZ}7Sa;b%sKo=- zF;o*p;I5xefLuc11}Vf{{$-C`8uowXWKFH7wb^wIK8bU>B9pU@@OJYs`qqY64ag2F zV$p_^*Bt`P2M>4-S>oUF4BqXDTy8cAu=dV-azqU8;`r=S3F+arKAtE^<8IaAc*Dac z-xD%ER$}KtT5PteATsIXAXrQ;z{UO2$uW3umJ;|j+q z#F(&?qWx2%zIE>l8RlmmX#g9)(>DNSb#q1I>AA3d3&_OUNPX8-6DH~X-;uAI3U}Ra z>i8(X7CMMY(QrDAL5<>MmJwkn$jYZ#3+EWGj8J*mvo%^zKrsZTDP!S_ovbX+ubel} zdJaGEc#g-vKM+Zw*!|w<+tev7}Ag<N+mt3*w3cE=0TT@Mphsg!& zr@f-AT%>Mni<{BxN?-mj0G2>$zYDw%#LzS#lG4SHj;PeaU-kz``<4WPI{lXCXIzbGXlbB_!ek z$CK``A#fD^Zg0>xM|(IM6E8+B48#LFG~5~ zn@ecLg|HbKkUWuzw+BZKVl07J5wW~+tk!{MGe+4UBGLKoy}TjkoaNgYu)!W^VbN>m zH@)1121qP7pLdUNjpgMgKZ5Z$bu4Z;FGRC?>qATg1gtn zRP;Q1ISKdYG$?03NFh)gpx>L){C{%6lJK^@e+tW9VG@z8<#F| zMiCHiC@I^m0-B3B!1c}1zHWnhxnL3{J`hdU&Qk%i-rUQ3MlYP*P+n6p$DQ8bQ~v-@ z&-jH~b=>oug7~q7B{dNXZa(Ud7pvvozlzXbphlGc07HVu%hPT1{VKgex#;<{@{=A; z{{V9Hy(kPkW7zP$7;_+a*qyWn5=4y<&gldi%8RWfvq)pwT+Fqvo-PN5MqhJgy~hI`~mW-jC;cx3-paa-hE}z#U?pXttXuhJb}q zA&wU2$*DJj*0uv+klGgM&T<~gn3bZ9a*9-pyk;;W(bW)bPqbObwr2DOK@1$-1aWmT z>E>(S#ix|KlxfYr$mNVMnXJyn@cIzAjU-)}EgtD+Frmg>?b ze%zw9`JS4&)|1!H9j4odve62IiN`&tp0AcI1&A8%h%4ylWS$OG@?N1{OSW7VnimvH zneJ7P7_(FZ*o_%xA{rJ@{0#k5h2o_@Or2Wt4 zJ||e&>Bg}uk_YH$UtyOPqgdr)iAKZ`Avyan%Ow=ng7OHJ-h^h%@G#yGeUb*-nlAEzhl9Q23@PSsYK zh1At?7}MwzpQv12q<3y@Rt$rjoTQNz$I z%LuU74G?l^)yXyBVG7hk&h208}eoT;6fj z&Z(ma49b0lo%1{2czSGHy)#Oc(jz5ANbxGARx(Dh9Yd>`M{fmNyUrqm>M`^$a=pc7 z2RiOAJDk0nik@xR$XYxFRFmdTw5l7R0X_)~LZrlkotCk>ls7Ji9(9{A+j2BbC)`ns zfibfk=+iGQHS@aM&^%(f~7(W(&EC?}$oh$j>2%}(d9wQ$=_LM8UaH4=P#xn~x3 zm8|0HTqL5po!iehHN8j8>|T10VDbL|OV4TQc>KkIcoJS6Hy(QW4<)-g!yCgYtwm3? z>#qh`zddY|u+82%u3nk~D!t^!kJ7u>P_2!tfY`%{1<*QvU$vp9Pjp@mV z76ui;qI}wU9gz!^gn~oJ`1Hg*PD4>_Fu#dxvMKl3ZOagpqRD|W=;lBj9$(PBQ|Nwl zlb%5I-P4b&(RsSO-d#ST`B_G#2_p-3W{pr16_}7w07^C)Ijx{ewhmtTZpts7vPbR;eCmvh!jkxJ7+Mv_}Pr1JW{XqJg3_HZU~hY4qHmf~Uq7>cR8 ztZoyMZ#*|GnssuGd3eFs$Nk?3%0>Y=+x1UBvi|_jUlI=MAx6>y_EdT-a3sS-p<49@ zAfVnSI!T&f*~;kt~{MS z(Z|u_@1wA@@)r&l9&0vASE<<1k`g^A>pcY{PECTSMz_=#(<|)2;smBA)^|vOBcm6R z-))g7*4d603^A3MlQ~|HvhI3p(85!_*1lV3fr+v$B3Vu*p~WRFjQ$!foj{G3gWo4 z3V<_H=W&y|XxHisYBs}#^OtqFmW4i&`zV0Bx7kGZ)f|R`MoY^fS!k949Z;R^WIiAg zmC|`XfrKO;rtNlu_$^eb5R&ytdUay(6UPdfV#uycdHeJ;e3!qKgSqGn>-H!ea(7=l-TCjVs7C625lL9s`Rv>byq?JMobxlqR%l7+08&dy zLuvYtFGsL$B*LK`s8TsH=p7}N<60)mG1e$%h0g-ebZM2RG<~_@ykVrYLP><}79?)l z16k+#bj}}MtE@6E%!+JcC+n7>X^Ft9#(>zewrkUYWM=6Z_!k9MQY+0stHwWcl1paO(m9+N%;=F*tMpUo@llhY5D;eb_MVcE*3W$!L%yyhwU zCz0xr?GwML`ooiG&;tqwxRW21je`R{VA#p?t~eFPgxCHau_@>e=}Q1o*| zSd_iXho-T4(YrZkA=XxjQT(d+n#@tT^F2he>5Ucz)$6R92i5~Its1YHS35YQ!)@g5 zxpH(d2Iw2N7v27E1qUOMu9EpFNonYXh9wZckr6=4L=rR5<%VUDA#qXaDtcf`8zR;j z8rXDq&h%=YSGfsd7^^Wd+{5iLPO;a`=iY|}7qgOey@QV1Z_T>l+D}*MDqZ3Ecye2c4*;1$|tJAQf9&9)8%y+4nn?72xj;Pj#3^;*pq8*2`z&X7oY zr_nzl^}MxIox6w7wexjgN*s4yW4%wE-=pRP=ashC$?Z7B;PJ#iWH!(f+025+Cr{++ z*@1>JgN@)s=Q+lnNln4(dF$*Gr;rm2L$Q5fo{lFyJXY>&g7u4=q(&c^{f{8AD;S;j zRixTZSbQXWsK9q<7LHz~{$HwE#9<(XT>2}s>5Dm*3e%={`PJLWl&<8TAzJV`F+3ST`izUKic(8-wW=A@z$ z8fO_zWfgLS35r6O69Cor79x_E9~KDpU8S|1$`!i!-q9@{z@{Ukr}KBH){{5RFW)*I z&R;s~ozMk{W-LO7RB^eJBEh!l;wP(zXE*?%1YJfM(f}hbA>&A@2G1d>Mg(=HAUQE< zXtR)!D}A&qeiIlTnlh7A2ZZGz6iD3Vtdh+1WAZ5zA2sAL7_Kb5#Z?jWwf_Lh^vh!4 zq*Fr)&^bLYogFFJ3!1`w*_pw4Q+*C@ffz0nbqIIdCX5tz6EKrrcsn?ONrT!Y02dmN zSaeO!Rc?Ds2Ewz59uGtgXRyeUoSA|yXSO>aMA!#RvIr(qP&n3;1P*4j29h^n!)wM1;a9J~fmhgL59-gnd`OvasS(C#>4-60@$6!#UDL)N&HLaiOL`}W+#KmFMxzB+GHYSF_kIkBp9WI z2qy%Nd&+mCnVVVtm&yH-(4#_>jAL$RzS=anCDIYmV04s0%!xh59mF7+tYsjTVkB)*5r1X++qXupWqhR$NvpAIMc&+Hv1ju1AHCyzzU8Y%%S`JOs~lmy#XO zTodzVkIa0dHvOpa8?nGhv7+!X(N~StA2%jDk&U!3ffPzXk_N1+i^V5@-+$e5zL(@~_PT-9HRM$hBn+*&w96ps~GJ+i2-{ zV(cr#&r2b0iupt}4(P$)p{Q7p&?+TLC#<$ZWBh~ho}DW%iv-Oumh#$@!9@39KO8JH z7mZzN{{Y^8xN_tv3tn;KIb@R&K@|T0n-CEmF${3!lQxyd2O*Sd7|rZ~>aR8e)ej@D z3-jfd;YjIua~k0%_7}S!_FB=aBKbrYml|ceH3jmFbf`$bhZifuRBT16c_~@gkp2%D z>h?!O}H4Ts2xQh-}63gv&{$3Q+* ziu+i&=G_xiiBRKYB$+N&Qe$!8rQzleg0fZNztb;)5qzoyfYps(Ug<=J$SUQ>@jP;r z8ftW<)ug|M^@DWdM5SuNNis(;N$XBAhADb=0pB6UPx@d zE)SQhBJkJ9;$-^hINWm8jqNb14K5jUukFEPQxcDCx9+cBEgGl%7daFIo?%#n7#uy&dDB}5pysu5oF8Akom@s zmB|nBWe%i}d`veB;(z{CooV9apa>eA;fsx5ibKi8ik+B+M>HI%!^Nmusqw6NSqE1N z9&$`#0Cm`dX7YwOecr<3W3WFCg^AMhP+23ovO73j$nRUAFLu-sbGbO)6e>G8)VEA1 zI}tKOH;ZgF7@Yix65~7DfuW}^>su&s5mAfPW^IOY(YL#ht}pgq)mhaK7Y1)xd|w`8ksDQyo;T3={d)vdqoa-EC|a(2JhF>kT3PK{Fy4k;z*)~RqZZ1tIglX z^L0|wgCTHVIwPtJYBk}0N6+yk824m`&K0f>F%n+CAr7d^q+p~RoNW{kak7#d0!EY+ zVM@weY`-coe4-8)B&2+v zw`Ig5u;9uTJaQK&gsPW~oT*CH-rNt38J3D)26WDi=S^;RYl%lA2Ao zD@q@gk?O@RsL5MS^VMHFtyLcDEnmy>HJ>ZL;$hh2t|rRLTe#(;1+g#ALcM4Af<1`% zyQg<~T$SL$%es{(tAnL-auckp^7{OK0he{;U5NPJd$Eq}aa7o(>UV#_cXxHv>n{_! zpNV|Ae=mCn!kcA5`4?039D7_fL^n&qFD)v^!7Ww%67ioOl2%Zu@H}Z51vPPH;gN`Y zkCo+JBB1QvRIAAb?7tTjJe1PZI@ddD6lS)5ne=XbW1-H-E*w_+5l{x@0biCx}z zm0!U-JM;W@J%C{2r}5{+ed+L@<%Qz!*_?`%hZA;4w?+;u17pJEhsX82 zt^yN_UcsXv@#e23cBu-aDb^|03vx#pMo31a3?a9^IlW#ngeUUn|o{KoCWU?qs zv~n2OV+4hB#1^YGDg)(#}6wGKZ3@t8|$)osy&j5 z`gLLr-&dS}12^F7ACX725c%Q75N!(L-;p8t3)f?oy*cNp#w-+4%h!lO)8>qE5hIi! zql@e=$B$o+*qELs9u#E+ku#4TCnk4t3!91F4PG<>?FIDsC8TM#QEI_58_SjaT$ke+ z?&`*pGSUNTYhRkQ+_7$iH`l7wT)72e;i+=v#=`Zu<&Todgloi$Ead4ira$`1m{%jWNiiTKv6M7JUMFC+`W?~+slH5xF9~e< zX(@hd%9(io0R6G!PB_-9e;K|+{8t4Dj67tRGIR2^6n?iP z=DsE`h+{J+*E)Fe@`hpsiy6-F;`c+3*NoG5I16675mJZ3i^#;aE|oK;_N5oJHUipf z$tyR98GKsW0MR(UEHuWSn+#j2X_KSk^Se#2%ceFkq%Bmui?crsay(fy5WZB^m3*4r zH>Vl--CvPP$~Oc#mVaUE&&bZ)U5vw*mo{H8!;e%V=Vl%k*p*#?4IIeIXMr5Bb?@s~ z^42KNoWJ=)oGLB4?wt_7 zhxxKlrs}RuKkfk%u&V&*WO}&;e#K?UuNT=wud`I7-Rr-nV+uTs+=Ucyi5!tZ&YZRY zNL)fe^+}#iH`atQeR){9qmkDZ8}=?($JXV;m|`w5FtMsZ&Zs!nHIIz1fku3){A7Ge zC-N`o?5oEuS18pwvo>lkg(*CurU{6xHD8J9XKhv8Smq#TzJuS7X>mhygZVVH?LHldQ;4 z9yYQuJ1)EJ;$lN0lc~&;D;`(mcIZAXSzd(|)y<1`HN~fLhiVt2*C^WaEg|zRIap$8 zpY{I$oGM;9H9)A-9|Pm~@qB2NjjhY7%lK}^<&TRQcqdKxlkz&Vm4|MC-P4ZZe~S0< z7cY}9*Vh-@<@`g##>nDXUGuR=HgT6;am9|{8FIu=E2f_yF|hKPKG*fol4wDMq&u;y z)zwK;RO3ZvgNsX4qvnb(2M>^!!#7&ItkuZdYKYNERe$FeR!nmUrysLs$KYLjyd#ukA;RiTb3TbnCVyS& zpRbeZS!VkTdi~Ms#3hF@2sz(4JBI~xVrDs@VM6ctPMy`3I!jbt$}L=3dD+*Jx}*w+D#`4wQ@gL$Rx649LV?3I_*qhbmV=cYviC(`O z*!xKoaX|HRcI3kY)KR0NL?QyC9>Wc9fa}uaIKB}R5xd|b-I8(kUJd=!LpO@_}JEM_{0Oa=`#ximt zly&X#JlrSfBoznyA&D>8BGyl+WaW9@TfgwB{ylBKUxS5#mz5`nL3stSV3I18OH`ln zABx@G-Q85Gr2hbd)mL@_`EN^_Ya%{v#N)f7_hSv!`3Mj^W}WN@#4&M`CO;61B0UJi zLl!q<-06MOk^2e~`vO!Oc0ZL%QXKa$`{VW;F>`x54?Wy>ISGBz;VheZqyc9y{kOUa=O9kZ`VF0U~2J<-aY|cXWkqRY1I-%8+rb z>hrv)lBVdkFP7+@nHG_~xr<07b#Ao(0F646+8WlW7U)4Nlse*dLJ+K9YdH+TNym;J zDLa!8wBz+n*dGj1i`Kcx5oKlqrbw`B(!{a>rc?W65KUXuz&Zk)QRZLxKMGy}W( zuT`@c2B^eOvi_LA`7%GR4OdNWkm{;lD7)w)q8q-aeN-d44^%^);i6FTzK$c}lJUu2 zOMXtn(aRuq9eLx}fh(~TMyh(JCEIEk<&msmbOtD@o02&>6t8Oxbc`DuY(n;XL=%-2 z&IcXKBslVnhRBLFD)tYnZb9L5F#g6TO_Y6nm>IaatM)(Eqmnn(&3ew1v**I-BaY|v zp`DM5`k@54nuVa;eyv=%x!+f<$aYD!MEN~{d6uzSBZwQ zvayD|&R5!~n{pYTXC=plNn*y1H5Mqf*Z>QLJqHtt=Q5ZKd_jbydAK5Hla492<2`#H zNjepqfLL>Wt}=&nn3T3IoC^;hsQP&jW0X=!8;p-9MUBgx6Qv$rYFF*;ljNs#{J8!! z@^xg%thqSViaty}B5+5zU{tZjycAX(5#m^=4m>e-Aqc5Y8eeWPuP{i9Bxz+0IR-$J z6gLdYAs)lZ%<(U9Fx>Ck7>-V4nIsZyh4>9M%Z-w18zSp5Jp4qy%eg1>iy&hR?AIYC ziV|bTl7cB$v36LRtFLpzARaNpKF_P}Vv%&XxTBdlw7$T_A9oYY>OWu24#Tr}&|^g4 z7R0y<%DD#JfpT&VqnPws-94nmnvht$cuHG|)SSf+Pw5wks@t%?1n3f@E#b-X;?K<> zchboYGgOACst*qstdKQZ9CeZ1_&m@z{yA&H7i7e7vl2utyCHmJmv^p**)|A-B1X=N zNOQTF(m7Pu69<%ChfV$&Fv&12KhUFYd5>$8A;Xu8oAu78 zvdYf-&@bbfL6Fp@ZogqDu35yz3rd3u-%Hx$YXFJQ?@*yxNX@)||7 z9JaT&HmWa{?=LCW_$Bh)-mG$R4E*uM8ID}XCOoRm_DH0kCc)ULf;@!j{EbkZlRzP+2Al67zy=7&)nHzTBmOWbtP@$B(d1lZ5v zB(^+;7E)%!w<$Lq@oeKM)MDkl+bov~j(8aip)7P!M_|fMi~}rq)1aO&jv0_M%K~6< z5tm#40I8knQn_JK7EFj>!GjV4kCJUEUyXQ-?6^f8VtkY}LP&D6eN3`s zCNeftVY%b_Qwc14?7X#qv*CDI0q7ug4Ip%(E^)0Vd2y$c#biiaaUnUU#EgbeBx-%T znFxS>!veU>X#l_B~tlNvO3{pSqP2#c@g!%%PI?(j!}^3P9r40 zXCJK0{{RfRU;B=AB=HT5*5S-iS^II4k%kq}F8;b1Vt`Nje{T!YffoU!s1DG}dF z*hamaBXMVS9{E>Jg2mM$e@vsfOH?uWp}e+S(BExSd2%Wz{1d;1i97!Q8r_|F%7ol~ zRv^a-k_U}1&QCWymHnH63{41fd-m+Mgn9_eC$s&QA6vl55r-x}Xhn`OLF?llLgiyz zu3IseJ;8)CNA_uBGLm0q_7pZQ-dNmS*BbkRWd>B|aif_O5dos#IPv7Tgur72>A=+H z_6U|Z;${rw;s!EB@*Hi+9BSo5#CS7!kb2*&NoI9^$~b%$mlUrJ)ez{SRYzQOQyclb zX$KZBbjM0Of#nR|xaI!t zDMA7JPEZ)f>2g0;87KbB6;7mFMku+NiHJvyORx7iFkEC9#E0!k9E+usEPPkFSPjm=+#ZIk`u0g05sQGVnEb~07E}gSDBNS0zh_6A{>vA$ z^VoHuj6SL3O%78r&C5<)PwT(nw9|UK-CiE{wMImnEv{F4d%ncqhMHH0gg+4YvuscD zlcgt!kCowbzBstp$}DMPbDyMf;`bx68wUnfiels$*n)4ljB)aT?H|Un<$4UCRFmZT z@w*&JBfV71gwXnrug(CO_8jaX4lt0S4$#{Sj3L9%%ETy(2v8L4-NiR%2~BCbN; zmt~t{6z_4#N2ID(YexGW)md$t++je!M=yEL89f`lk6x zB_9XtF{qi|pL|=3EeuuYc?LMGh~q1X0B<4t!CbM65?Exmja_%3P>1-XN@xD(E51$k-M zh5lyvC5lO!S#(}Qd=`O}S~^l%?vD?8d^M75Sve&T%Pb~Q=DOquLSPk?N%<=TCc#K&WcmlA@0x=}c#(X7}%#=753d0lv%+1+b~c}XB zN6D?)Hkru!AcXQFQ130Il{#B>A34Bfne- zq)D9E{{Z3$V_48)a#O@eqc2Hya}i8ZAUv{7U}1`};Bl4+IEE@KB4q-N0ddmDg1AxF zkGfFllqBebSJ&s&=Uiy{y5m%#udgh29A(Go#;>D_%P|1#XL1x!jMjW_Q4TZb=;AoA znakGW`lHE+B2#9MDae^hTC?NwaAT7`Y6^L)l(|~C)AG)EPHc?4e?-Dx16}}hU3mPl zCUWFvWTFn4IME*>`vapQG>g`o9my0N=7pP&`eef9BZb8Ay8_3aOr#SOa!PpPD#wKr zxbW)7;JmobE^LH_Ba}iiJMG;rMMi)vv0TesytNN5MB1xkhc`?J`7x?DL*iy;$P|YxNWb%$o!-C>xmVO~8EKy?+$C=9> zX`FJ|3>!2vo0Su>!?60LZ z*Gkk+_3UXzU)69Y4A1`npG7G4pQdMZg2z@B#QI#Ht=DD0wFYK8icrY-1e(2GN05ke zvN!CLEug>oAtOXcJeUV8jAU0Et1BdvV$qxI*&nLpmZWcQuMNCi5)6+_-*jPElTDE> zHY?J46l_sSxQv>Qm=G4Y3M!%mg-_3299Cl9>n#P#(EPYAP#+PHsoEvSX5CpRSXbbxwq{qs!+Q zK^9cl@%7FL^E*y>Kd*#%c|Fvj1?=;oo)XeU}9P(}qi3A5D zosaT`wuF%YGou<6>*Bd_Z`n>>2w#OitcFV|%+3#+_dOJ*DkmhTsRUSmj!a@8D7N_R zCmNPDm=*FY{O>$!1d zeFG(pj4`>S;eG=gq$qN7v{C12{=JZ<%a~Z6KlP(#NhzcbMpPn0aN}~(!6CiJ>npJl z!J8gT92n${MHnT+jJ#oI*$4!37obSW#gt;bsOu)iJfzJLPEobDF5@!f*$1*oKgts9 z{i9i-2n43C4@Fqw=AW3B*$g~NpUSvBiSdmuPC!k@KpaUpeR5RsA0MHT)L5AU9=RBp z%NRs&W3>X53Zmg+GeaSZq6t32!3pHS>GgqN1gDjN;-9C7>-Hs{6G$81F``~FgOqtu zG+bWs;cnswg(noG;_{-4G%_Q7T|IWcurtIBbQ%&(ECxTu_S%cOggY8k3R9h91$4 z2yd1?GZGE^4oOK7%t+%7bPg|kpdc}va)faDlo5=UXh)BVIDq8fBP_f`9y63)Bx5(y zDQA}vSTQ7w~bB$i~T&9S={~MJ5cnFqnoUk1kjeiy&f?CmMP1<3F<;n3*z! zkeXPXHh-hAj6JbKltG?ABq$UTdee(BNTY0ATWO;Nq5jHF5(H#YW0M*A3OLW5k!-#E zkB7+Q5Ge!N+-gPjqY?Fo1H<{O7a5BA%)FGb#Vp&@aucdSDI=4G$TFD9Swdb%Dg}_>P&9K zKNNb$FiD4#K5iuI&_oXXJb9|=@YFedR1;kOp-9IUE7=D?vF;{(kxV1*%$FlW90^J3 z9!38EmK7jrFPdD7USlef z`xyn{eBwZpa0jYPNQ#tVpqdeMal&C7DCJH88e{9S6X2slrjeC6RgAdD1Y;oO#038U z{U84TIL4iZA@zZ`6lP3_6Tkn&04Nav0s;XA009L70RRI40000100I#ZAp;U1F+l_{ z6CzPEVL(E0|Jncu0RsU6KLIC1;NW25nTKygAa?;6F-&nF*Fb@TgBOdz;*Bv7hVeEn zHIj@|Pl*^IrjvqjI-ePq3|ONB6@ZM8(;(JH25GDfUMYi2acD?sI4EhZr?E~34g)4^ z5D}7$gk<1^j0m*W07P(7#|5M(0|e7-aY#K53VH@uraq-O41>7s`-tMvfq;yZV7yQo zYXc_%2R7bsM4Lofrag*MdmiB|#g-wDh)YPo`X&8}Wa6)CRF*rFHm`;zc z#!gFp467TgSl|?7b-b3-B7|V#+rURsQdr_etx=*5D+4ljJwiH^qZiY3YKA!&9qT15 zQQTl)yqmyD=rUOQlJ9zejMgO6NeyP5#CpZ73=9s$G}1#5jGn_1Yat=y&?f^W-G!IL zb*p$JHPDu9SY#vY1mco}MLIT=3|;|2Z6?s11O&E*x`Um>hB>4Lnn-IEg6=V6gVcSJ z$JQww4?t3TlY!9SBNAom5sE}wNJvPQSa0z{R6f41k1-Olbia zDa=roXh%}dq}V%;%K@SX{uxg}(kRHuX}lERu|_10@)#*8L%4<*zJs-5uzD1+`f<3fOzb^0V3dZK$>N*a9T1}%^;{M3N z=z5fgia^AlmIbsRVBlnZqQm`%>O6TR+>L64U}j^40RE)AWA$ye@)>j)lXxdw+DmyT z4B9}zMs2p+c~i~g@@=%7+DouGZU=nni%8@vuOP$sET_n&gy=IgUQN87NRoBUqb21} zHr{V1lYKa}&@TYc`xe{Dv=-ZKw%czXNB-@RBL}6Wmr;Yz_CIjo z^w@yn#2*mF-a?a?hF+&`(*hBH2{HkM$!|tNy96>B87*}k11_Vm)gz_1 z*gG44pk)9HIez9SIy0JUE zWq`>&Z>$5kAD*MO)Oi#WjlQ^rKd8?(Y#u)Uw1vB8ZkzF$`r!*lr2! zplfJ91^R+NC+d@$Ktc~fj-&+E|7x{?%Kx z9xv2`?mn0L{Ylb>Hra3d1V($Bgim%5<0!M`B1^a92_8DXk=(JUYxV6G&`?v@jD#Zv zDGB>AV(xvYM$w}`*i$Iev;JMcPmbTy^7TJY5#zbFM@MJ$4(_kv`9|F~%%XZ~n1Yzs zQJ9>LlE_9*3VzH`?`FI0MushxrP~IO#~YE=2^Q5QB{=8B{E?SXoJRyBq`Cs$NpI~% z4(7jPddJ4BV}+DsYg25E2Xk`EZ#af~!1#%3lbL$+c^{T0@)86d#7~F-U1FMWPP`gvST%9MfHcEFPy+M~e>_ zv1uqqH=}y8Ku=PHKwYu`ia=`-=?Jk6Gm1ibMhP8`1I0gYlt-mIl&=+E0ezIi(HB3bGe-UQ`5o~^#4NJSWB@~KBN@5-< z2trCu52y__ingl~4ZMa!@qlT!lgK=tOt&w@oP-83v<8rj0Mo^m*e>K5CD(o3^3V>}fuHq>zJ(37ZwJQNN&TA0PuaH8nA^7wCf~^NwpzRU zVm>F-U@=dqV}Xiw4xtjfLUd@SQ*XtmP=N7#VXHszZQue@kJP`AJf3uk+em*TL(Dg9 zo=DAgJ;1k$A+8RLQLDZ*Wmk~H)Qxczbfkt+r<|E;9hqEczw;zKZMenUnU;xZEbu%^ z1I;oi7Q$2>M6xB(k6LN2wU|fe5J6a(X-G#B-#-wZ2i#IjH6`(jt`HFBpuyBW;$&< zKk1mT)f@zQh^Jfto2x_e+vE#Lej7`@wGE!~BL=z{%Bb->2a_-j`5VW_5X;*C)nbuIh^%9xuV z;u$zBU>eP*?H;$-3{nO-v5u`LK)`8$)4R4Yy{TvFF%MsH6UJ7~JcBENM(&QN6wMk2 z-Mvy<54fX%bVxXsz3-)@qa=g|yAK%PA%s~0>lpmJ8bi7*kas;~FCp9Ul#!@oD-QHI z1VDrmJ$a&}%fxu?{{Wj0Bx{OJ0oIb(q-xf=rm;;tQql$~r0b#+qjt#{IwC>k7E{+p z^2yuWle1^ng_K7|lZmJPyS}9;#?kmlVdJ)zDcj^zwN9~pPn5zOMLbY2B%BcX5gA^u z3=yMr+VIY-ULZ@A^l?!jT;P-Q!Nti`j$gGa^hPD zP>ln0`i&SL>If{N31RDhXT^f1Q z%T|dt#{{zRm#8?=(XN2$liJeKG3mz6*^{&M_KmIqvXdmqp63={P&{Y4OJ^}Iv1oag+F@{yJwGYXBl3F%fp5 z$k1W7A`)cw)=xHjnu%{+e;rW;zGfLb4Dq0J~DK)`&&43m^l;6vmRs z)~VIm2t&mb8A+e|YT9d8TSr30I^$B16Sh7OG)Z{dDBHmQ03Htw9xu~zz7LT^v6tF%AR_@rUW|tvS%vMfVXT`24!ya zNH&C_IwRDEfEu-un?WN+-*xBzH7KdbrM$7qQR-N!ux&k*vTP@&2Ip~>878}%w ziqYGb>gh)CFc~~_L~=v)$5+(pn3{AgJVPywz2Ed1{U_u900=aMB-0%o6Bx@x{WN|V z?MgV&y74k?_0a{{UG<0kRVCL>?qOQypGCB|6bLrzR3oWTZ`AlCiR8Y}tMB=ry{{P}5%th=2O%Fa^Fh{C{Vc z_XqWDwzY3s#nvnaGpF*Sa^J|yRBrHj1N#JS5XySmPs_{owi4Al#Axo5I(5BRZjfpG zy3urzB)^i4G7&Mn9YPMRpd3+8jaY03R6|c3arwwNvuzgMPsOqF&ua7tPNZKJcvjp-`YAc1c#0Kka<=Gx`&Ys-mj^T$!r0mR~va~fXL|E zP4N#TNH(7zSiU@q>uCtc!9+(UU`AR$2-73{zb&_)StHld_)t<`6NBRyEWDm+wk?RoQ|j3V*+;H2#2_KF zY#e~g7&r)JucL`&mQ!bw%LgSG0f<6bhG!!I8J}SqBuY@0CAOKWPEJA+f`fu`aX1*W z7@q|Q62oLNlw@Ed#j+UDjCRHXaeM*v0x}s%@NzK;8HO?tn=>i)81iu#pA&$V#VGPq z;C*v}KX77|dYesW^-Silry&PDU)8Pq}3X zu1W`k;@J+@N%R?I1)mj4SwdO*V4*1q!1QrnQ4WV35#~J{RCoa;23Wy37@S)bcJJ`mq_gIssDzw>2sl2ap9jRCvytKw zoClspP9X_SPC@W88>J0X>|_U|wZZ=YpgnUY z+(Ha?&zZ!sfhkKUIR_!KMk=488OX)5eu4T&r)csrj|a#=;@`CRlzBL+euymOV%Ry4 zC+XY#m-V}Nu+)4f5XK>X z-aK~8`t^)M(+oUbCE!~QHEUiY7Oa-FknwoDAGVq!Ou|M(!e6!+csw2~W;{=*VTKSG zVIktf3^4HwWA=Z;LBt@-32E1YVdc*FDGt|ON?FOsYmxMTV=%)QA6xdn%S~wX{m4${ zuGG#U=_8!9+amJG1^G`-(2Dsf!15aUK1K)qGo9^pEi}>XzJ(pOJT)lazbSTa_bV>~ zUC9IaX6V8EBd;Yms#B1UMIHuE&_7pp+P04Ie%-1L-O8I5zpbpbsIN%8AcB{OE$YKl zK1*b^K7&3AQIgdkO2|J| zH9U%IA{*_R?qb8`*$3_F$AWN1BLO4K2`biLH&>++S%mRoOE|j z5bWEN@Z7^L-r2d;rpP@o?$IMu2X3Q`&zNH;6;Gg)w!kc*2=!(V+_!f96lP)Gwf$pCF+rN^lTRP>&}!Y+a&^ zA9l1P%U|WGmwQ_~WDecW*cBQz{gQPq-t3KC-|5@Y`=g}ZSZWhQ5)fy}&4ve>jeB#N~BNgqLU^JkV z0iDaZe zQZZi9f?e8jcJwT=g4^~<(c9cLv(8d{mRU{!VrJ5OPoX4&x3^9?AB_RP284nYymbh-joMVz+uP&XH2M&M zfWx3k@Cq*vY{!DjIFz*8waf;etSkPV2VTo#4fQEX3&vZ2FKi}^AVMr<+unkUgcos~ z0}PY!?pKMFYMf81ryP<`iO(ikXVklUeBMKBBYvNh?}_dEvWriGl!jf8ubo8AIr1qlfEJ}Fs6gXv=CykoAh_UQI6rgoK!RSp48<{{W!y#7oBz@Gl$(idbcsXVfr^$_s0V zY=|j(b_-sUoAlJ^m`Z$dL26~;R=O8@n#f09DZXl**+LRfS{xh_fQD14z`^k_@_D?2 z$(C`;B;)@8+ie=#RHwJ~&aHa!z5ZTyXw9@DW>pVxlJ<@+^$+E9X? z+O_E{!(CE5903Vq0JGo(gh=oXL4@j;<%vyQpY+$p`g%c1a8rPI=n~8$@S7uY5pZ5e z5}iag8sOrgs|lzsm1UQjFAv<8l5G9L{nwkWC&c-fLDwB(#j$qnBNYDtO%ILod1K{} z*kUqYgctsETQe;R3x3ny0ci7w?!41yoIKu4vWuqACRt_CA>zh}dd#&fAZm;q0!wFZ z&|2eQ{{T2R-x4JlxPvV%$*_Qvn$}wl3LByl8sJE6xKeb@YcEOFA&f#%f`R59u?G;I zunJB-;x{KHDaPH4+q0lTaewliG(%y?Lrd^2V{kUv*GAAZWd6S(>+fw$> z8kcdD;CLutwWDzv0Lx=6{#kot5voIauu|5#rv#?iqeAVq@w=s(hobIQEbMMZk1$)% zSZMvG;MLpiAN@PpmL5=>Jeg%RF)e=u6o)_?+Ho%+$YeaVO`-dZOU_eb@|Tz4H%5s- z<)g!kTBj8c7lo0xA`A9tox0bjahwbQhOXJs2L~JDGTmdNCYebLbK?*77L>QICRw3KWXn?O894^7>xGn*KH z(8a{nwkH5R6cm(diE0XT4Yht-Jf3Y%iT?nE&Ie6k2ejhS&bp#y<~l*iy+>)@qxU7- zw4zx}U-egS_O|ykeQb>y8N|0u zpl5Qha8ip9^Pcf7b!t8yozj-U?XnvG0G44VMs#Ems(MBUEkXHhGok@+QcjM61hw9` z%Y92)CeZEN2blc4!ff(?V}FQ>v-dyK_Ql(5nx?3sYqp)fIS7Ueo00+fw z#G=!SHYh{;9m^?UW#LbOZ*zaPRn|dWVWo?gUDd8p4yj^ zZ7INiXRLRUF$_C<+nZnHwEe+-E|EHCbuA%$(`Dt-y{!u8eA*;j<7=~dyyG@`W z79j;TCI0}HzlKrXNS>Ha;JS9$hm*)4rr>t+37>|yFFhKXwXja_;rp`vz+Pb&(Kcrw zJLCCmAfq<$1iT%qn&@8MA%N(R*gSM#wBEIQM&)g)W*VJSHYkLd3GM!6CQ3gkA)O<~ z%5^|WYb<_J0(I%)dq9NNUIxzsFl6+LYR-=-L5G@7jZfv26c>^&qrJKO!gWB+#-4dV zMWLqY2qM~q;H4)_4P8Pvek_QM7$x|WfSQC?PeA_9c(Y{IDiMTK@oDu*YSc5!M8hSq5}WVkboH!Zd#0BcgW5(7Xfo z{u+;m<+3*PmVg97>Au(^Q?nvNU`0R=+muK+u7$VxB`uDabx&zJF(H8%i4XZ`b#2qodjYWcY8aGb z<+JdO3?roI6RTh}C~7-nM~puyN@_bnM^cfCkKr2N6coSz!~i-F009F61Oo*J0|WyA z0|5X40RRFKArdhIK~WGQVIVRSFmXbGQb3^ukt4A)!D7)+6mpV7qO$P++5iXv0RRR+ z0^xD&#~I@adlRphL_2g?a1`<>#}78E9>gi&9N|SAK|Ek7IQa*N3$GYmeEO+SRX#>I zSo$dlbVJo+l}f#g%O9h2rllA%ut(MJ(>3Id)2a8c(K zRZhh`ihQXj$ek4_#z$kPn0V8(K3xz8%N}HQBd|QmsZ>&U(O0uiA_r`F(5dpY94b2# zy%7WhvljSyBd2$bLaw0P68P8j@- zJTX|TRw!`AVzF4PRx1^X#WJ`9inP;>+ZkDJ>DhVOidE%vcUDWjEVlfFd2{l#S#8~h zg+D-4Y^zGAH{UD4aqg%GgbY(y-ut`Bxmt`?=S`3RQDo$u(*VJ(k0w-hr_7Eq@-7TN8%E=M3qKJl!Ria}+ z72lAJV`?iv`XX(Y{>#KQf}a%T!8D8rS-EekT~M9&!kctn5t2t@zHL5ywY}X@ij-Z7 z0uW_^#EliHqr5djd0Vhj)~lFunU;O5(z7z$2a40CtHXXXK$-MX!5OVtwtdL^eCE3Y ziA;WRmz;YP;(W34L$^g<#BPhRZvuANczI*qt|qEnAS*KFFe>Q3qJx&^v{o@f_7hXh zsy-f?uS0?!B^Ug>fhYd}k)u?{Q<$DzzHXj$aIa#D?LAY5FplchQ0`F~ZM?fvij&K^ z3dVy2QdBS-M(VAiID}kd)l2FE4P#*rV=4_w7eIf-f}sv> zZbG;MPsHTbfsPZ3j7%Wn46akkIw~3}40S3oMiiCaRziumbyICTs{a6^{!5+o3mPKb zIws9E^a#+vRUwRSIwxjX%Gxy-7esVixNe2-hM_IJTFNS(5{!&fGcFaPsQ`Mf4el9? zD4dsE!}RiL{)QFS-lP8j%8#Nnmo4QD8Z-&F0SlA7RisL#70-Re5>*%ZR5DiQV5M1v zYKvHN4@DJ`SPFBf_Zg1OcwzH(<6gumE_0@<`Yd1~8~*^Jt!L#8;d|BHQl(BW9d5~2b*P2!j|%*CylP7*f8BY!#C>+*JH9BI;>H3P`ZPv?t z&@#r1)i*7o7&P-j5*7;%6C5ZO94m)n;t&c#!-fk5InE7HloU-BL0+l`93|$vAk!v& zfV-_=RFZd8)m-@K6#m+;`l>8>LZu4AI6LjL4qTI7+ z-Bs05FsN(lu$9q7(}W=|wM6=;9GPfEN1}*x z{Y-G6=%iq*eReDF4du|#B2nC-Tu_wXyBVx$#q9a+pHCAh1<5#dYB;lmr2%MRLiW@j~MVu83$prCj1+Of{dnHmL4_ z!Nx8nJnNz|kK_bvNUCE+GS$ZbpbLHL?{c**XO(s=tLQ zD6(RmT(8VOmB|u!#YO%mX2SrWo4TSyr*|h1EgPZ%gdJ2Hf+t-OYqh|l0p@?vH!DGh z;Se#|UAk^&=F{k=<-DxMp6O`{xnZR0xiIL6AkRRp(O0n?CKa%#gsLjauNm#ruu(Ec zQm$zr3=nZ`c)7rHErC&W!*I3Y!H-SWQPiL=q9*_qnR?=_j}kN(d(%N^fQR)@bGgbE z!a5SDn*1I%L9aDgENQ4W6yZ8iICUyY_@Q#juW&0X=!LH#P0EovtWk=r6cDggQ3`b7 zMS{Ufp44{XijBOcmk1|*72z17(g1XKmLw`*s*B>Oje#ek$&3Xjbz^ZjllviFQIAzI z1=?<+XP{E&V1xq+tZY_?V6`qf>Qi$qr0J?9yPyHYqC=_^oO}DJw?r)(r`s5&c$49n z2+m;-RjLLqXh1wuegOAQ>J+SaQCEb8UNWP|uVOe8bm$PIs8!(#iOu%@B8YWW0;++6 zv7P5qP`3t(AS(zt^kZ}j#3Ps4YO-bJM|<`}w*Z{uKo|Us;}F>absf||?)*yN(m*g5 zG4EEYjH>ik3Za%LT}fUbhX{bw=>0MZx)30Gg&hRA{l8Yf2Tq}ntEcdQsyQx{>mUL)#s2)0D@ z#Yy)B(Cx5C+AcBgOt;zZF9=o%LIk#P*>n97baL*wPq}jlJFKU-F}&E%vgGlJ)NZSf zR5LP!gNsznQ%7~%#yzY&)5E80F!W6=X*5BiWTBx@Rx%VU6XzH}1K170rmiJO6PuaJ z-ir8M3y(XDdy{F(6yJBbSl89mg>fL1?CjbwXtJ_$+xq#&fM~LEs%`p`p4?VpKFgsS zeXSK@c0|oKRPN%rfXr#tIOS$NlkmDF2NObSn>3swMRQ#}6gXpn3VoHglU~gc(FMAL z!ZuwvM;K5Kyep#(c+&;4=4yN0vbf+@G3&=HdlMxb!~A2EA_M^4mEBPt7&UWf(Uo$*5#~DyiIZI0no(6>x3L{ge`H(LV_qBWZ-@xU!@6pk^U0jr7+kFaP75vMW5zg2fx4iKua zs_3E3Q4+C;K)1cRtAZ|N$A-mPcv{*FN+(SfLY^V6ag9~fjRMZiI1E-}VufJ=Pt(RdZ-cvrsWz2{v~)#ID*u2@v{W**G>3!3ez&~>Hdlc{{Ra73N6LjE6at)*(&Qs zV^8H4wddcW7Xp3Bxqw`PfH+hr;ddBQJRwg3!Y7Bx=YVT@iAL;7 z58Gj>%`)hf%S0}51-J!lm^k|GrL`W{`A+`;(ve)UPQ3Dmts)Hs`Fl&UV zmdepa|@%-cnnB#OR%H z0-CJ4TMeo?k9B5&64w5i)k#&uWI&*mGm36!(E`KBeu^s>N^izVOlx1MBST67#7kx*0b2POUv68)@ zI-mtls{KP^49NR;Kl_MPu4b~L6q9Nq=~0MMP;3^R(26!u>%%TLS(&;;iuR3FydF0uud0{jpHQyJFn@J9 zxmDaLzQfcbd^m)22XPftCbfbQ%W;O9wHFs6M|7Y}lmfWmrB0<)Y>XNx)l#bGnrx^b zZh$7LS~*ippZ@H()-1+$G)Mi2u+ z%*+^1Duq7I=vrb_yAqu6E268hsw|m})DcBV3J428g(|IdM#!9(1|F(>GU)cQ>YUdT zD57;jaY3vST2q7Z(&cPIqQa#}yw2TP!w3V-)UB1GWlwj| zYPqraSMbRFkhP+i%Kb)y{k2#RL|g}ZP1Xc9lPcDf8Y(wV1*XvEu(lW6I@?>8%fj%( zW$`ht3B~f#0C%|p)*BsGxw@w2w7tQgcd?b2THDJc6#Ta^G~IPk&2bRXA*kg$1DDK7 zF}lN8Ov}ssqCJ*`14YhpiXqE_puOD{&oXps!WtDdqIjgK)f4+K>%$m52N;sSUnCRp zHxzSz(Z@IY4lBbgwZnATGzxO+ek||!56p^aBESy(|6SnqlqZ|U3d-DAK6iOxnMW$V;_llS7tR_JMOM;(D__FqmVrtj;O$0z`xCXom@Ix@{HCln?d(%DkuMw#VM(Lq@&nWfI% zG`57M-B!CBP-tj+hGGt(2DsY?O3HQJ+SH>^NZeqW{L6E+4I>2^V%Ey$!5WYn6MPrN zCg1Y1VEa+-)i_Bl(Coez&H*}GBw%4w_uYaa!_gnpL&}M^Adh<0<9wNwfsX!gN`V0O!~^U>DCU3xw&MG3loe))~NKN&8~|pF8a5X=c;pF;4TqV8<_Cdu$FqS`1!bI zg%b>DsvOmduL+L1>+ai@0L?VH)Bqsgxy!dSo4r8QG36@MEnE?3Vk!8kw}M@4Wyr>dEZ zwL={Oa1eG(bB{;}&Nv|W>Z?uZifO2YXgt3WMnH3aa;~Jj$yT^xi(3Q=)kB{h70huS ziNF3Q~vi>op7l-!%-W@iwk~CR)PbJJ~9N^jlV0cXM**d4a*5^dPKoWyC9mHyvkU$?$t`nf0I#&od%P_An@*$u z0N|euCW8cSGF(%@U}DxNmM2Wrr+8m|zym?t!}EF$Ka_b>Y3ojWx+#Y4ZM zc7*B?<}ftj|HJ?%5CH%J0s;a80s;d80RR910096IAu&NwVGwbFk)g5h(ZTWIAphC` z2mt{A0Y4$4Da=BK<4UER#cgpnV6c|mzLfzRIg3xB6__y_mAyczinJS-UOGignN2-) z@dr`dOVhb_wJZ3DMsZafq&r8{ze4!3=vD+=3sTA4Cel=HQ8=nvR3Ogaw#7twjZ>{m zRy*N5OrmBD#dADEjY z%mkz?S(ULN8)dqsN=Ts%p)v^!ET=`@AJ*veT*Zk9QjMMlz~oJ8GI z2oRMs+-kj9LUQIKNm}9r1QC&aCRk1&w21XC1E>)w#3J`rBU+;J%&5)!vQk#&8H=L_ z+&s%sRL-j{Z8pQAP?(iPFo9praI-hLhU4)l!Ra)c5`j{!!fqx2E*w<#Npo{1pd02{ zf+=P^NrvH74i$Kaj{g8aiAcG`tbGlEW-F~kYK$_$(Gu9Q-i=Poq1=85{6R=kiXq9F z*no1)mInKljxH&#m&*hcq3VwiP-$}r`UOm=Y-P=`LY+m)^var{0TM1_f)|J*)FT5m zF5IG%?TdhbYGlNtIY5x4JxI=2u(*uO)gH{^850tE$_B9#^(|p3;!*DKF0!=B^@ak; zh6=G(7=t-XAJrN@Ic3c$il&f>8)s6c=n3)=BEU2%rmz#pCF6ugr>P(+HrZgT&5i z7H(c*4U+7mmx*M>g!Hu|40oEDgtEHC$Uw+>mnwrwWW>0#))6671w+8gmmHzsrGjRu+L>g@Qw>0?nPG(#1MUXH%L^0I;>&>>JB|cadPFY7 z!|5xJT&ponMiE@Vs7ix5}6X3C9Wkj9Z<9zK(-=Ov?!>>48vWC%w=AV^5L{+ zJk!-%GD~1cCSijBx)x`exCC9b6lt`G$h>@f&JJDL2$silana7S$kFUb&sTrta( zVRBr{61gxOy~6310+4~+Lx*yZvl*PrYcB>k978K;msyrY!>AZ>QjslFe6|SN2M}dQ zaV}BFn)N%FG|adl1H`h@Dom(_G;!3ptkNgipP7I}7QoE&a%IpfiIIBZ2t=47`bCP9 z-d{3m1majrEtfHOguya^bGgHXU1zy|1W2sL;}vj7`tUPrxq^-Bm@yFUC5Tll5K?1A zWOrpLx>x5ZNUB;~#Lj;c!m0(q7$*k>@J3fC**+pvj6sy(;$OnV%ULHqW(H@4dUGX0 zA+*yI&BmlI5y31F^moS3rw>NGRwV{vn+i2D!|^h%W30xa+`ZUO1a2cKVdyxCdE870 zwsY$nj9hhNm^dQNXZ~fQqbzqQac~=%GPV)ou`s+#QBw$NH>j&CGZIUy4UtX6TH<$o z!bjGvF`eXdH<^e+>xj&y$x|y4R45w6SedO%(;Y#=#Mjjh1ZF^pmrO>yLKmADmdFEw zWgWxDDpn(gA?=n5RN8J5q8o7>8JU#I3_|5*1fMbdbU&=RG@V905h)x@2r-SQ2*m3{ z1*vfzs#oylQ6X;LfMYY)1C87ya&b{Cu?&o|uWr)9H)>S`dBChfi(C=5_c*btA5g=&lw3ekqY{P* z)`BtVj0-E$!I0F+iB4_Y%=8(}X1-C$EyS&bk3qF@&)gg(!e>imDw2poDhgozDNjrb zg&>PbSjV82_sN-&ZYECSk>!LJear>}X-QndgU0OlG5}(EOwduW{NXl+<}X_1@tS7w ziWO(1OqUH?Dp8lINVv@4=JfK#ZV%2-EU zqzz>w>J%YzI-#l=nUh4zpGB_zqp-dvqce~R$d!uat8*JMFs4XTscSKNRp`Q$&g8L_ zV;qGkbSRk%h(lDuA%rf(wE+?T0K;jM5{YY!UMxVkg7XHEGcP4u%(!@#iy_3RmXjhS zrc=zxgujk4zZm876$dX6vP*7-I`kMz*U|}aj4=tZ=_g5)Q7&MJ(qto0QKNH+y|HbMlVY^a`GoMj zOV3=sd`=^Ya3;J%69lD}<4JSWS@kh^IF-9(R%M5{k5Z)6F=d&FXnIAIQM-W!JrqW; zg-cWGcEqqfzZm+cX;il_ApkIm7YmPom`eaBt889?DrY7W4Mt%V&1QdEEA0aNV+|af z#?lTRNbnW?hcPgf@hYelaRM&D7RnS_`5i)(y^g0>V2~dDMcwGY@>Q+%2&PwiyU1<^i@%6K@XV$9K-Fgw z6o(+bD>H4=4@9_K@J$S!8D78lq)l!e%<{z@&7(CP(#$kX*vbvc>`kThVU3I|LV1*~X42*?S-QCApI$_)QqaoX$eO*9ZZT~m;D|0N;uWVdg5g&r2&gbh0I79J ztBZMxtYTD9xnrndRSFc!;W8Bo;#ploPBRLLqcc*i!ZRj-n7Dz3_o=K6#c@gtBoS=` z3|Ex&kDHc)M6vb;%0I5GcIdI?e9NMAb|j7WvFKT>k(vkK z56$?7X*dqZ`CU1H$o1x2;~-tiA!I!crR%}fHwA;ZS2Il%R6U`(J!M&Rz@}S6P;rDT zIf5-QY?Pu6VkC?*n2A=P%nYVv?hRZj#IWaTCJPT{zEo7p#!ev%Mn#n}JVdg6CXarz zhoTiO4+xUvVsE@R6A?pl4j>l!$d1DRe+*lFw0@uqK~cJiR|~V9%=%wZuW&TD)x(rV zJDO4sbsF@=;#r5IXo;pn0tAKB%MeFVs#GT$Qig7NJLU<0zFt`GuSs^WT@)S6P9sD_ zXyk&oETMB922y6d0%=l^%<&K|?jo}7M)53Wj27|ouU>C+$1K65N82aDm&&h zRg^8~7I1ottVTT}d)9KyIfeAO@DKM9^$9Hj3Q8l|2bs)95aY6z{Rqwi33)>hR}h8F z6)9Nt8LA^jBBpZeKpaF{$y|Ia__?7;pv*L;$2W5WT5YTD5Z>^Nrl>~G;t;D%6qk4x zdu3+Qv|TTVzJ{vE{7jDC2-B?n#{Msi=30{A9PTruD68sb#ijvq1JDEyQpHV1xRjYn zxu8Hrh7()$HL)p+EEnj3DlkIzIfT8H7A1^ojmW^5Oy;r=RIQK{`+|WsvjrJr748v= z2+A_oH-`+rF3rd7HF-zFr+0dk@5tKB5YS^HreM5OSGY92vZU3uqnd?+7fnNKRu`Ly z9BS*I`wyhb&nJLlE3SI-mlu^uL*h}mx=p1FgS5)1P_=>=k5a!`vY<06inwMgUhZ(C zk4G?*;y%b0WF^tkW~u?{TGSQiK8b6Yc&7qrPo*u3@rbn>5ygzF&KqJ2l>WIYRfD>t zv|8@Ei*RkuH#OP=L*`+aelr5wc7L2o_4Xg#L=;?txrLFGQT=W@;!Dd(A zGVp+3E8&lpEc08=*w^CZyS^ng`3PeVCpdo_=qf?2xU9?kO@P*%!WLGg)H0{WCmZE- z`51?s9E@AUxojcSE{R*zp{NxFZUv;J?UVyErU{woh19I2L}W%r(Gt4s8ZJRh}V_gT=?!K10|?kA1e-m4hc{841eZ>$|G?` z1;@%%s%3Q+#dA8+nPwP6mUR?;5I?FvnaPgg0mK&uV3n~4%(`Y0>H~1XQUo}rWE{*a zz!=i6L1`tHLmtr8Eh?iK{{Vlei}+!ZIFZ+Z>gB&4A^f`fX4KHnBb!|h@I9-6eV9yYAcH}U{q)&*q4wa zw97k8!RU-FhL5E*kc{5+q==rBLpYWi!b0@{iWq6nrs+{YdR(GYFzya1fpYCoSc5hD zgG_|s!4au;xK06qA!j4`AV9L+@5D(*?5`l z{iFQEu^vboxn4PxvM8&fYsIrHw_T)L}^IWY8(jG0lF zfeW4srTB=+kdn~{Qq~6vlMu>B15+ko!x>bFG3lBh$_aBbH!x7geic_Q>(nV_Dut9~ zZ_KTd^Qm|-*lHV5zSk)k58_)#CHaO5_^0)N#SHv;j5KwpGRVt6-l0cy^TYt?1b2e+ zL@y)6u`*^Zs$NbbhahOYLgxXz1aAmHW_{h83FJF$i9_d*j)L6H<|vzRVgO8{opBFn za4I2EDO4PxNV8k^J*`KQT(gOXq+u-LRBmV=Kv}pVApySO=>jVqHw5GH>v0GNgy@R{ zZ$v@Dz_K7TI(zb3JEa;#vb&k{_vM`~m*}uywQQZWjUSK#thF{FpZPwD*vXX1e1?;Z&mw+G|BC@sMy|LR$gwX_PMMnarX@~)~ z95l_&z|=%5^iYUM&1Nw`1fV6Wm!@n7GPWbK$Hc>OvdBoDz%F(~iOkF|H5m#n#NV3Y z*w#VaM=1{yinGgeCNXJ(%u`4I0A)M{^F|tM%&%NO=`m!*bXzSY4-kwsTagNty~dbT zMKic;r<~_2Icz|AR6Mzwu$HCQ_MsIMk~201np7<}uA*`iHy#rBe(T4BSS|5O_*}mgO@CY|gIYXfZ1$ z%W=*#3xuOlkOp2A8(mMPM7wso+XXm_GlZ-SB2|dhe~4ANfA@nV(&2x>8o$VYH3-V@ zALPrr0{mqd1%dMnbZ7%V5DhZzCDE}Kr72s*5%e_VigA<9j+N1zL${Tow8P5VpzP@x z4?6iwy6mW=sETW;YTi0r&(S#isghBWPufut01<8>v6-_q^;xTwvavvOFHQ*s7c)mP z&|>KVa|}>2(8J8*?me*IQy&1YtUfB^;^Mid40>Pi zf#pwdK$HozRLlyoVS+75-UkWdXZ%VQ0#s8Ss3xl1T-WHSVCiwo$}$D54-l0E)_n76b6UyQ`-4JPjE@7&ADzb2l45LC>X+bjFb7JvZZzO3^ZlEpju zg7S6JDu&Mz#?$6j<hpye?A=&+-KKpyr8bnnM?2# z5{26vw=ou<2RA%@vbf&d#hajB8y zu3Sen!rmZ_8jd((>&hrx&`MJ?S(ud*yNK+J;RfB1cC`Y_S;wSf^C(oHq;rhQ3=6o^ zY+M#EcLfMZ1y-Lz6^UPIPwv8IKa*?(x82O9*QJ0Ev59x7-njS$c#VCMz-*nxz1me{o5{p{S>|uCFo(#HCqeIhk|JPZJWQ(K>|f zV4TwB`bzU1gw#taz?7XJR`4ZLLdH z`R61GZFm>;j9@LzCp{zKm$wGB>5nF$czYl;S?;A*2xIXYOe5L zwPGW&OI&=;x_#UFM?x^=J&Q7kr0($oY{Hik4E)wk0HxV08+bQyg|uv`rGVYB7^4+) zgp&0kmz0FxiDlj|4Fr0^m<9#huL?7y(@M7LQd2D_5SCw4VtQb>N~ZXt;_DU2ZZ63h za?X^+tWPiI1pt{yOr6Ts*5lXI2#0wwN#&SmW9~TDGXhfKUe*P@M_nsa!@C z?BT%^tj*Q^I+vf^>NrCeMLCMYV{aAF3aT5#q!o8(P!Ou6T+l@S07-56G`$47Aj|W_ zF4(jEeNIzUM+DQ9mz0A{@hqs5xV+x5u4SV@=f}({6V4#CAJnm>Z5YvLVq)5QLYWcs z5hSJ7C4NK_-g94-_%gMcf$5D4Ah41RK$iiE==9iNMKeaE=cu$`-FFDK zQf(a=&oaI&RHL!3;|f>GTyw*h!zmvVxq#65M_|-GEG`fc`L%Y-@KVFI38Hy|{{W=Q z*}CH0eqgY43!f*%y};`Dm$qAIoAocW)1B0*V!x2MEPmFLaahsdfdCvAQo$_;6;i#K zX5ukxlzOYB1n^WruK9hz>ot0JnLz?2a5$7;boRkba>dkd$pL8+Ft}<1dcNAfxMd!r zXZPwU-d-4$4D&4%Oqs86l-)+EoiNqe9CZa)Ikp`(240+P}W3qKL-QcrO#V{W8I5q{3VJGj(o>z zp*kjA1LKJ3N8tqAH!02Fbh$?B6ym?;B|`jjGU+)JQ%7Av5>7;LZuw#xbeosy!sS$< zqd8JOr00TEokbZ`&KMrj+XS;zw#1~_OwVY$VmrSOb|o1*L0oPb5Qa4jk}a9DQAFA> zP9=O!b9E}NPUQo5U`w^}4kK3qf9!4z?t6uO5x{0;+6n<5;XclOrTvo*U+(3AwWZD7 z$k5EbXsjIp(450ATXdKrB9_)r$Uy}`QkM0s4Burp`X9_a%lbdepuqvve@R+#Gs~t_ za%qO^=ktkle$uRxaABOON1>B=&*_Q0rn<0{vp~lp7a%?_zNHO^hY2z)*d6)w!_h!o zWgv|a<4kkv1BwtRD&~?^%9Q$)$YWg=>1a00D$#&C%pqcQDmGXW8hOjmxffD^Ev}-p zaB3ipVECGZOI=I_q2>VesCRHh?B}^# zjvHs8&3;Xzv938!|jLk3i2eyZmqEkzQm#_SYVmT)X zGb;L>MqDe{`JQB;`zvuBoQF&t6d7!$5~SZh`8DwQ24t7#X5}}l)TF1$7HEQ zqj0W|T-cxrk?i#{jxxK7Vp=s9P|6qwp>0q{$Lgw8y$WbTT>`6@%VxsY05`+v*4V;%d3IYK5d>~ z5F2N8Oi(WPIK(@*UQh|(1_@jP^op7>J`XYFnA31uCRY09+M+l$2RL;S&LLcyM+jI; zDlsr4X=ma6!Y~qFSk}^}y4O=$YqDIgeL|%^7|pZ?6JNE&w;r6-c9W=pQOPEZpOgR# zXAHO;6q7b;6IG@e%e}nB3vu%ZhbhS&sfGeUxGy<~V^c$Kn6|pqsYemC>K5u8aQ^_p zCHb$i%Esx1(J21JM2ehAM@*(zQ`p6dyu*VRD9n#gLep?OaG4p_$5NB9jT!cYofv~T4>199 z1cjA4VwFX!CJT7#Q7w6lU_THPwLCxuk0_2)kg%mNH{3^N*_Yv`~MBP&5&W?mT0(Hpo-EB8a+1#8^z6F$(Tzh-*y83V~MV zaCUB1*>bD_Egg93Si22E&5_&(XdX}wQ01-AHzZX6L;CqfX&UH1b* z(WWgbDa)BmbLI-gG_MlYI7748jK^8r89zz1HCgdC*Xp>}!K?d-ysQq6*wg~lnKL($ z_#ryb!1D$MYpRXDefo+qc+i7WAl^Yx98x(B?n(8 zABoWxMdoBm5}y8v^EIl6roYUr)?7^(LgRiC-cOj{5x7eXB4KJ|GdhUSA!}7e_&=DV zOaz(7QO|#fT&ZAtM|5*+taIe7S`yVs+DjDeW#&7XL~HR=1Z&P`!@G@2TZa&V{DU(o zRpwxgz6pMzA_dyia3V9-!GJ(&DhxU<<*$Hej!t6i5xHvex8fB?!V}5TrPQ7>3d*_* zQx_$>H~#>Vf=b_hVgqcbe=MMQ8)n!z-dKRbh+Ma(-};X_L}gxQQ_X+0Lr`gZnP#m7 zrWbHH1zc&B#3MXV^6 z%axWWYl!F-h*|~2v0a0RUhs-W5!jK2GBSX8lyHD5{h&3k%qspue+9>bw}{Ki+lhpC z#s1;2ZB1cqZkGhJ__<;7d`U}14}x6hdbxa8oFXp|G4LMN6`wE|iLQvX7o<3q&BHdt z0b}=EIKnpTIxC4U4n8Mn3w~z8HwFw*#Rm+molRGVOM@&>2&=-kg8n&01-bFE@+^! zbr&T1HqDCj=4@IVMN&UgIWyF_)8?WQ^HJ1l<_XEvI{Sd5UvnKx4!N7KqfE$)jY?kF z)*y&}b30ZRK%sgiFh8GfLe|*Xj!~(4D3uTz53a^+QRymF`*`0AO1(lK1 zFky~iBJH7yiy2DUTatns{h>C^EY7nBFv{1{i<(LYIO!PHCEwKf<~61&;2=Rt7N9H@ zR%E3aZrmDC^#JPWpZrXcvKT?`Dv>Qz~hGrpj&IEAO5^@$i*ZG&C z%M)(%;#DHM>7dUPLB6mVGP+l+rplBNj~?XJ4xP z%)+l@sm$Nd1m;-{m|QMZzmi_H*3_yT8TojNa)J*NKgidJy}ymFpd&PH1{E!&$;%yv zuR;hQW%(uJjk7o+bpl3T^oLr0V81E|IjZIXmWXQphB~nGSLFZ*5ByIvJ+P2znJ{XW zcBZ54X&MOm(XU305|F#xM4>x#0NXo#rokMw7=z3T+QWSijKHdhgDQ^3<#~;EG1}gE zZdz>8LKfE*GN8)SGZ3XSY_~p{sdogle~EZ|T(#6jD%G-&1X2hxg+>MvyJFRpPb2dI z7dzzN5m-zS&tEYjwH{{X0;eg(Ij^!b|GxPB< z?PM1ad?N!yR;N~B#0?NjhLyqqS{}FJ2!&hhzNQAn^zkhSvbvYzZZ6mt#Y{_?>z|kW zLJsJeFX3}lNXC&7sP9&9Tt4ADp*IhyykicncfMh6lS0}503u#cJ}SHa08k~bPY`$h zqnJu^#yzz6MfsO-TGB4;`jwDD#DKSJL>1YDb3!k!aZnz?Xf#ZL?3i31s5Xnari=VU zSBxm5oPU{xKv3S5p0xR#AB(=gN2Rs_ChmvliW3+89e97d)Z+Dr;b+B}ldd{+r$ ziD-5A8de9kZX1>0m!Awmu5d8~Qm*)w<`u(rEd^+@7ioG)mEPc`ms7T^gE**?jO8AG z7?vu9i!SfP7iI%&8L?2V<<-DVGH5xAM8w+3>+^~x)+sY{w7rAw#hFUuD9{}xpXYm zIfrtXYq>*V5#?N50QBOzp;LsK3ke=-=+-CY90k!+9AF285hrDk?0iRBpdDG6Xu-QO zsg0_xDW3lTQkupct6$7gpvRUB5A!UtyKfXr>pBKuE4Qnkmx$mh;4ds=3TS^(KyYE1 zPcgK*Y-g2KwijyEX9%Z9ApRp1bxhmgGXfY0#k^>gF0$dL`2?#vvj7oMiG}-x`k-G{ zM7SS;oW7*vhzF5Wbp>=C!TWr%cCb7uEBwy_9h@%M7q$!jt;%KCa(OU%g?p?JU*|M8_#c4v&{{T?2Erj4d%)RW7w(bCxuy~9|QhmRu z2vq>KTi@uE4C*y!F#$Fa(?%jB$gx#t`GGgl8y;m7t5E7*P|atE5#Ser1Do2AoYK3%DQl7!zFLfEGZx!`GRN zKZP-iwJO28tCoco?QZDD#8w;^Z6@qR$nB&2%FDTQH+lZ^6A86fAJh1jlibPuFowPz z<9YL#^NnY8hXR#I4^Amp%oJw)Cdgn7{I+$=;LI4jXSjHpKRxOO(EW*e1z%Fi$!d#j z=;B#m<=$1>tiW>A*$4d06wnpCKnx|%0-5|UbIor(#z89HbpHT+%ESStH=$e89ACsm zjoi_KkZ4V%r_{9=>9Sk}RVjYq^3csiOyP=!-F(jw5Tmx5gAc?HGZVtJNkMcnS1_!5 zaz)zxb(pJYM<Y)2CXylL6ikH`#kB)9-R!q9_ zGNWqJkKOPzFzDG#*uEfgzcjNh8BsxmHWsZ)#gMVj_M`O0gv?;lfWRNtVmehY7#Jo9 zxmf66YGR$|R^xXX;i4Qo$Apd4=f?RVY9BMDXte=cQ=w^j~bVkjdh8m?mqU0U7v z+^gBWGJa=n_qWny+b)qT8Fgp5;Omp+a%wP+Fz{t(`umrx5a{X7-eRHcX+MV8dk+YB zzGWqbTCCW2RF=vP!~dO=KuTp^uA;7es_ zmyLg!XhjWTJLKVJ)T(n30+dxiJWyaJSc4Ns5EW&U_;D;sFDg95&d+ zMV~M#f^&6KZnI!|&!7SYnYF2^+v4IHhBY`r7T~#V9j;@y6>?@cT|7jg#zMz2t%cn9 zg4v;v`inlDakX49O{9C=qiP994=|SV#5rI( zXPcJYJk^pXZi&XCtpc?vlKZo=I+$c@CNC2TK(XBL{$@1VvUjpu?BM-Pv!r;izqO9~ z8QlV#<9ucz)+hhk01N{G00IC50000G{zrG;=avyyWe4wXqI0m|AB!^M4x!ILp+L0=f--%9)Tj>an&u`n6`ZKos!T&m z&;vI-;Hj;pQt#1lop)xeRmLW#iGCw^YqK1Hv2eZb+-N@UlCGAdYx};xYMHw!#EX2B z&Y)V6=@k)&3?=LT06TIY`x+!}q~frrC5Y4a(b)D8T^lP?cD&ZZ#RhZy(#+H4W{f>R z@0^glUj_&FKO{R8(9O*_Coh3xB%Y{e*fCBYNP!h4g3|+P%&|8dbs0~~2(Gx^2;yL= z%*S4<5H+;k)e}l<{O4%mf}XKC7$TOLu3D#P)2PFZP(6Sr5|iL)7Tr!ZZoVe?xtHvt zTEMUr!sfCTea=(wmx*ZBG>8}txl;!caD{Dw8kGsLiP%0*wqQEC0AIDXcH+P$nT+sW z?FY{wqEPI+n0JDdMs@UQ<(A2}8QHdJ4Qzur?3a`q& z_>>dGDOQYURZ9ZvkiqBOP-ho|7+;+-e3>G~lm@2m;a$O9QA2CTcW0<}61N4B^}h$5 z6QEoC@k=Bt<%KOSb8mVJ4P>;1h8Pf`X3Ckma zaW0!0bLb^ma65w4mq1xMtMrZPSa8BBC^y__%^SqdP1QvbzT52irMtZqI#mZBzE{^H z>QnXI)JEEaVsO0q0^;^&8Z0gkmiOxH39*4Cc8K(j6As|%0MR@&?T$g&X|eLtCbPu( zUUf|5NYK)qJyw79_QSd2+wL>D60I)Z<{~3n*G$u)dXRs~pJ7$>`I$3TV48D1-4+@X zypWR|Hf**wS2eyiNW_Il*cU!s>Z_z#2OYgJrba%_%SzxJrYN5jI1ruBIZ#(4oiE|< zG1bTFC=vjJpIc=w(4LOG1n?RM54^nzBQegOoN0Fy5UuF1?x%#_#jn_j$z_M!CyF>c z(h4~py_r&@T)%Sb1^u*w!CFq{r)0Mf!^OXyJyX^@}nk@aTBMJ98nT;gwCWVku{zfvP z=cL~WbG$cfKP)t5kBeF=;lMh@!wdjIU|&V#&D$gNxwl<3ZZ*;G611H39Am?~Ain6X zHsUZ@D%g)HwuP%lM2LJ`0fz6VI+&u#`qRB-BTmX|z0T$58Ahof70!O+o)2{ZcPAi? zbFdu$02!={`|uy{s~j;%>>Mm9A?tgDdP**~9I=GGaCOq)Mof-OE9nM6^$s2=b?deh zRE3O+u0-524L&tTi6cSiVeu+Yk}jK$oQ?WzGj|FdM=Q#=MH}xn2QY@EBQz<jT#}{s0--%_qisvPk4s6KW#d#IvYzELEqBnZz2DjxLJV=}lVnV{$YZ=WJSc(&waRx4T&6=>UVB?8P?gT6w+TM^{6 ztk%Sx7c3wnk|wsxrR_TYCFr3y4F~bS$?7tJOy0yIn+*Q|y_cL?k6?L#mC&q?wKz(@ zp0)V@00J58NoF%iEkvv{F8~!8Tk=WlgC)*h4g9%bQ>G(p zf1?J8LFS!#xJ0yZ?IDFY636EpNti!Rl+&RLGagG?P@{Zapy9&)^_|HkQi-Pm#sOe_U3Gxv4WIhDMrR*6l!pp*2+2Z*=%a@B^ zJwh)bj?W*aPx^q_uweN1&cm9S9bLc|tBY{GOKlF^O9<8%s4apdojx0HV#$wTWH;lE z%L~Qs>9RKL_^|IIk!OCHKL@s!Qah1*_79F0WaWE~kiA1MA0NW?ExP5KaQqe|k1fe_ z;Q5l+S+}XLa_!U-+m-G+4xIRBtE>0nA@DQw$ZRd&h9(3j=bI&x+sX3c?juHV8A!^W zHTuWTFiRPBJ`pHm+XKn=$;$=b%)^Jk{Cb98w}>_fhWwM}{TTQNa$kw0l4B@bnd}h) ze25S)EX$C@7x8l~`Qi&KoO?XCXP3MtIVc^mu%*>RkJXYxEq4ax}B@$Z2%=c(<7Y;zQtwb|z%_8S*;| zIWX24@qP__>lwJ2H{ii%8xiz-@K0?Ev@qb0q@RQFJ)3QrE|To~K23zS*oNB<$Hs2W z;&5vroa4`d&!YF~l3Gjf(U)!0;OkR_$=|0IM~pHq-4~Y6!;@`nxK5vrtasoz!;=!W z>4PEIBjRjEd&ubwn>+GPiw(W_8>ubJIccAM9P)L%__2=6r3N9f@n!y&1UYovS{Vyu zvhfc5J12`X<0E?r`<8tV#rjSOE+yE4A^A&*(`FESXwC^Bv$u=P+_1;TLd{*9^FDbl zAFCOFNqJyTm#A`1PK3S)h_cJcd}PBN=gChn)8psB^XcR^-Liy3ut#SOU3?x-Nh0kY zNwDSGSY&tWNRCgN6FwhdaeKUz2rT)QL?K|y4~90QB=%bJ8+iQ& z&MzC^f=H{7d{(9*m*E*#EbERUx6u^lwwh*v z(1T()_?sWNL_|!MNhPufKFe76_vdD)5M=Vw836Ct3_0+3>6b4MzYy#!h++5;a`JmG z%bU&Y!{P}n*fPzs4~P0^#(p1njobI&W%#=}d1lGL3l5d>LR`7rEse~~e7 zXOf90ZIfW^uVFFq69i3$*}Eb^4otx9hCcxb416v_4*c6>wpp9 zc__BPJ(=<~5AA|`om(t5hFcFlD`nXrN6;Plg6k2yh`#~J)9?)WM7_ND1kgV&;+cCU zMT|Q#4~9?q*xdX-A1oQj5w|^r{=oYaiNa)m3#LqoFuM2W7E7$6WAyLHdv$slY?Hh7 zb!Tl~Ep6n5*aZ75e;^~4hw&ceiN@2-v9Jx^!I=7M4{tVf_BI7=6HU9uM%i?fB zi0dv|YIw&k+wpIAQJW_nJ_LI{+u38|_VePzw#y_aZI!XvZJv-sL2feLhI=t&+p)ML zx?wRXJrJ{4MoI$ObFGNW9gbpLY%kW^dY%_Vzdigv=syU8O!<~W;|+p*w%D1n&RN?z z5of{hl3*9*VdB@DoOf8_HvUbIL~mPtJQo6TVm4c1Fm_Uok;q}}KS9rlKF7d)XMPWl z*RaCQ$3Eq`e*OIV4}xA>u^;`$)%gtch<5%V6J!wb<3R54{{SM6uA#V#Yw zmzYPe{cp)mqZjYU-1yIv7IWc}W_gAN#I5$;1n|l2hhj<27hx&?05i0(!ZUwKdi_u1 z9g;O;aqv3ZCYLgTCSKaL1S2|*n4QBp@#@GX_D_(U!&tIh$+FLk&#!Fw{{Rzy6#& z#yYq86K};VGV5qVY>vk*<(3TJzaAwF;jh4BKQmynBHl-_<>DDVY=gFZekJ7Hk?{c( zTM2;(+#5RHV*+t}M~B2p{vzrIWaG$CLpkKX9@%B;S$p`vsoK?UP&nE8@-F2YHM{BIbTi1u3K?m%k6!qpf^EQallS<1*TC)PeNgp$bVaJ@~o z{4MN#n=i?a`tQ>BEG|%d*^iB|-Xos2RzW5i9?LnFfoYeJy1~oKk6SpmBCFFHWWCvw z+#|v_*@JC}@58vm@zD%+W6z6c;Vie3%XJg;FUXP!i355607(A;OfH;&U@|=U9`8|b zaxDjB3?=FttMY#BU7Z%lU-^Puj1XS_1QRQAU7seh?#m$9A+~1Ef(?}si6F5g<+ct& zBFW|33-&h=@?3!jC9f>UAWC4qLKiZ86pw?XOe84Vk0ETCdy}6Le4B0-AhLn8BpdL~ z@+6t~$ZG*3m$qaUS=GS+%d zFYnH*$fA>Md$wZ>M&*l6B(zCve&6Dg{E~IQ@lKRJh!_qa7e#iX_ z{(>~q*!W#!doDbZSOq^33!@o&e7o<%MnnNB#h(KHmnPo39qpD*tjUO%&9uvVZ_Uyk z6Lj*!Ah`i)cK6-69FLC+eOkP@?6x}@vyfL?uoEV5=UnoJ18?{9F{&w6!0(A&zXkVzg<~nmQTP*nuH$PxeGld$1I2!AApE&5ZYQH zC`cH57WVFK-b5P7t(3YNyFcdKM$G-$QloRHJ}g>O)doR_%wVR?e`Hpq0pCuMKr z?szSvpFV36!;_EFOV#9j4`q@}7VAsQ{4SPR1NVlgTsEZ(jl8GjZ*Q@<-%Nx3XlggLa{)_B@Wng#8}JM$Y#C04wr%#izeB68tQ{$zhA%J{@)+Gfp-*`U3_zA&^u&()*E}5te0&Q;zv-+XQJJV@v~#lMbceJ&3M>zJZ`MX zhB^c;CU5o+78Vfnli_fl-wDvVOIGRRJM2GT%PEPmj+=AkyCj2*#@Mv^u+M%H3D3 zT#w83H-rO>m_|%OLs)r!h|ZHBU*gENd2hhmC|GadZ};vF*>>;C6v#3^mypL(BgVqi zd)OYN1|!rbq&vhT+_UNSfspg#vbGlMK1~mS-MijilFr*+{{WK6bmK5%09anYJhQNg z!tMB#yHPtq6=T z1CU4H=PkkR9gZ@Fbd0#%^FF8cV0Oa|6X$HoaAn#*zax<>X2Hw5VqKlYXK;^8moU~@ zo>|Kkm-68_jLnBxCRkX-?V<7>nIbTGzXTpm79Xb1>NZ}^P8;G!LJamls7?|OTlm{` zdW;>vNInLv)csM)Touae{WF2fvLWXbHn zI1-*0`N}UY9g(2`TseENE}V-Yat|Q~VFG)HJ^|Yj{{Y2~<<)+xV1wZN4!Fyb$0LT2 z&cICmpq>y@wsL&AO&o2%@pHSF^6)_hBsWquU}eTlWtyD3k%a#MiC@Vexf%OoG6`4q zM4hxfvG`sl441u!fo9A33OffTKMus3_CHYgyvXe;{L`=|E+mL63|%D)!R%|haKODI z&XxuaHc-PQN6E*xb)FGWWx=o?WzI@{jCo+|5xtjQNm+Rg`~$`SIdBKEw`b6m9F1X| z2>osbuni?kdH(>o7obOMlZCm)bp@Qc zeg}@oJ0N5uP5u!_Zv$~}&fYMBSU-ts)rE(Vr~$F~N2~INIdzE!i%Th=34bJKZ$yVU z`v#V-FuXftS#mJ!<0l=0(i~Ier)e&GXfJOL6BDs{7Q}-;lOxQ3CtUV4$^QWF?Z~)c zV)0LxON1>Ki}`OMp4jMNG1B983k(W!PM505v3k08T{#qEe*%^pa7%eEB6xuI_8pXV zNO8$9M_>TTAy%i78HMK-%iH)QVr(3-&^|hSs88>&{w(bEVQu8eA0~Kh32DrX$RmgO z63*S1AV6jiW$lz&E%7V`%)fga0I?e|F>C1`ZED$eQ z&q>x=ns5B2^oeYIU;tSQmH@HMAZQT7f3b{X8w};G;jbn84%m*^@foIe>C2a&$TMCq z{F}f)K|X>5lis>E8wl!y7DC}DS`go48%L61X9nU5^t)w$oW8R%)?dV3VNH+Jek6G^ zv?Pw)xj5q+oVESD1mlCr{Jgk7v#_!bTWyXYBCrDu7|S@BmJWaBR^Qmb{gYig1{hum zJV1EHXDm+q0BI(!9!>6YN7%13ynqmg`{HXjG2 zl$H*&m_8oB{{Xv4ZzB#Mz4A<+yr(c!ndPz(b^zM&rECsT76Gmtw^ zuq|!YUHI&{_7-4`NpqGpxV4Q{@mmFKcd;$EA0Ie172O}?2wO1-oDx3 zWI`IrOdYtd$wEC}^sS4wunWnvy#HD8hdqQ zW0F7@Hu#0ep4&YqquU+5TcSVy-B+@BjtJ$Mev4`edL%p__875;#|nQoqU`=nz&q&_)_6wwk(;GBj!Y2B(r`QFZg|fFXVHqxyX0s4PHNCCt+k6ZcFi-VB0<%W3kM64#C)x%Om0X z_6UOr^2l6c1iH_*j-$ z`L@Figs!<@ci?WU!e5M$#v7<`lYS&1PMCf!LmtKEJ^9>c!1L6g%{Mg!0dm9~Kk&D^qSI$ZeE1;SS5P?Blb~gR;+o^b9c4 z@@p`^ZRf!KmSy<=0F{ToJ~B{;;z{`ae}nh>U%E7vpe5`$o74Pp9~fc zZ7m^`J7>WMW#5MNKY21{ZTQ=5vRGWq%Y;r}i4TAg$cj3q3@5V27 z_FXLFK2{ShpC+y?p|o;vm*H&Bw)|hMlH2}vB`ROP796}wgsV1N@k?1=C6f2!Ww-0> zN3kuQrR=WtKlPSyR|sQmCOY95zkA8;kZqVkey?qg!}E`f*%UHEC*s{%eyq7H{0L$n z0cCG)uOfOzc*-OEtlv)>3e8Szb4}$&p8GHEmDRySbAj#xE1?vo2{60+yf48&7U7PkMdnU=2 z+iWuUSq=LvB)or-bLD^=S;bXu&e&-@x`#mfK^nOgRksaq*ma@t+m8?Uw%l zhre^+!)!4uO`{Y0E_ObaJ(gbD5dOu%JRboBET@+d+a!asOG_l5$HH7#4272| znA(RVg2{6u+ugjIkdg*Tzd|+D#Kv<9J_Fz}0whE6$7R;t_vf!C_)GX&%j+1Bay^)~ zEf$;puDqO%vKhK;tg|8e1HHfP2r?5Wgj^PO!{Aa_(iN*@mO=E#*_#kpWWlp$C?$ma ze^MRKn9&}k(6EC(9o}!`b(T36bKjCd{gYtJyM`g~{Zfx0hDrPJ^8K>UnPsF4B<_44 z1c&*c#PZIDYC`e7R(GD`I*+Y*=4P}WtJav{4DDb zzvOGbYyfTD__#F;VmluREA|YAa~H77*gR~X<;|lB+wwp$2J*Gr z`jj@^x)&|QpZksJ`#aFFL|cr8iT?l}+n=9VnvzU~o%tNUVm@w@$KhQ`jHS~KMtt~x zg}13;@mj_-??wWR$pbY(?mS&pKgU**&F zJ@xse8t>b4w{0Xl41dRnuxy#+wqxeckv>A-&&cY=NOR%wm*XZ8gR|$(pqaMY#kLLJ zEb;-Vn=u2_J(Uu7_vBUr+FV<+`ARl;cl;@Q8biMc7Ej1?;~UgM0|5FTsre^25&jJ z@sdljWA(~nxGrCXFPyN;Gi0&|x8b)UX&+YOJzUu@oqRiFYkqsS+2oytcQ z%u5MNcMYK-GW?qaNa2&PkXiCt#PV88u*y*u*xk8k$e4o2wtSX0(aB|%z4&}@!Qmu( z%b-m12ze<@B)dEL8gJyJjIe(v$-fLRj5yvdBUZ{jPxTfrS#s{VJuV-Gv%-Cdd~`!> z^pR|v1-tN@VYXa{ZH4~;a@#M!+cHOld9Lw}{8KcT{ev?%u9^YOrE}t{D~gF829U28lK5(89p9C`SPUJ;dL3c;RMNvc6Rk{PaJ{YVlDlQ ze$29Zy+JJavXDhlS5{kTXCjhSILYhD38b`=aRHI^%=R;fyq1UP#p)cqLp^dc@ida# z*z4os?hGZ9E&CYO0ptrP{{XfB07zf3&=`-OS$eRyCH%YW-CwC;2a+89wIFd0{B5EGtk1?Q z@xM+Xe`BP35J(H$IdLM!d21tb;XrhFJ+UyyYhR) zftE$L0eq zjNsebZH>dQoAET*($~4`k;H#~cyx!QqG22i50gIpApTpolP+33lJ&Fi#gKZDRh@c= z`!4wrgzeZr8*ka0zlFME@awFIiQT8|xV(nl+(ovBK6cu&yt+0lylD?j!(z7I*iDm` zocTaE^*x+v?V?O1wr5Ljpwr=Di!);uY<^51?D{0Ofd{K^*U18eFfZvBpOCx}Jh}@!$0Nz|5-AU&?6k8zhVfzSXCEj#V9g5%>c40LbmU~GO@T$9oZ*cRHjy4aZx2Oh*a*h~KaDY#orllo8hoER}g2Z)6>A>^SUF}qh_TYR5ZN!OOyjg9yX zEbet_m%oj+XUjdFHd~Sri(iex;KFe$o;Z-1y|Q5kmx%WEES9a3%O|id*&JpZ_%+%=dp{3W2a>04CP6py8PRm206mv~Y(8uR}Ehz|b%_}(o4 z0QF{@kNxeOb;RHq_-8H&(;^+{OlvbK;)MywT(jp9lQmN4`2q-9w z6cGtgQ1tuoi|hFhc6P3F&gXu|>mHzZF+?^b;g~;cwz1Fyw9vC^9dzfiTifTd7m*r? zDafC+%>wDnx79qcZ%J<}3gu`1T+p&W#Cj98NQSMyB3V38L#AEI7OYkgD?I((>t?Z! zK4;6z+0C!#SSa~bo4q%c#Xae`xSr_Y-q%|wYL?)T`ilQ0vOTPrzTTt!+4~EziDjPA zue7;!p2aiX^UPWC4d_q9w~N~{PmCcnxt~j3VqLN;@=^_IRwUWBOUC4^n^|Po+Ei}d zx@yWbTKThT6q7-&`W}T*CJOb$*D) z<6tG)fJb^B{;1oj(BBmU9$A)m{{sj&JnHM#3C-F2!{GTJ;0FB#;PX@h_geQlm(^$4My_x4!$Qj~%BCzmOP8f8vPG_ByALAhj8g5$a2c(w$MmieG(lUm zV?4Q!x&FQ^(&aDw`rraA{K@oI54J(;1aXy2TX`m)usBqfCNvhSUctvU@z$veuov;U z&)BSZoAmilK=zgI_pS5c!GC~PKRx^t9#}t$x;>lcympBXQs*-g&M?_*TV7 zTO_$&&E}%Pj~FAx#hvsnr$DUA&n`{%I^r~v-1xe#%74ME-_l%ehjzB-@}-?T&^{BXJEdZ6^!cP^Xs`{R{iuslKc^6uCn*K{cG<KN;%T!{tHwku{2h+C`t z@gb$-)h~6;={CN?cep;i5&&WU3t+wy|Nb9Dz`%=*9lWmes523Swg z&s_e#IerDIAyc7h;jSCcX9(Il8BAp;p6nWt>-jC(a-Ef@^0LGP?Nb9?_}J5htd<^s zjegZtM{Hl12XL2eiB=lGp`8Xyd>9>U2igmBt~4&N`tgpb z5vvBGsukJ^d#+4e=rMtnd9SFZfg9_cL#bgGnO*?_uyFo)sZ0yr=cA*Wj=ZApZ(*Ec zU9=)vlf+gXZK)RBp+mNRr5^O31>YJbt=UpYQ@i-mo9km;&Rb71tsS^siLLfRj$z4@ z%O0C+*a`W!oHPl2MF)q)rm;?|t0i(y8_%v?7g8|P&qLW_#;L+olhL=~r*mtP=`;*P zAEaA#s`L3X7^?Ktg*8#9X-aW`#;2~cZMLNn(^SV#Ceon5MgHD5B_||-0fU&ydy1+C zR#cXuM=M?V0HGVfqE5+=nWWiP6zTXvjuaco;)cw^T@cVR0oangK6bjw2c~ zy{kMSMEiv`RYATH*J*LzuxIwgh4v7ZDQ?!HhGtT*mB2!F)RZQk9&*!v`0M04L zuxr+6+6*u!Ci5P%y?|2<>)Q-&{;uu-xRpm!(LEVlzhbdFR`r=@jMZe$$x53}XImSF zsCx6v013aL?koWJG6Gn^1SJ^8+bzMMBS+ChR);b#m0CMf@<_aXE51oGT`{5fgFlcv z@ANhg&8Zv3Gfd@_7FYm^8`R14bEi=b;a5rQQPihXdnT(J@UMoDkltPA#`IZH+Bl?{ z&nxDRI>N%0MZv2J$7ux_v>UPv9lPTbSx-saYZq}}Ol$WLf%(G=$3z(EEDO@WiD3aj zVowDc2HXiJ6yb^tnB!nE6>KzNyU5XQ=y&a`TYT#aH>4Y849;FVJ;*cuV-PH5 zq{-*9wBL0h$2g+_NbPSinJWoH1h2xRO;*bq@k*htAP$fOugSnN{Va@pW)l~TKzQ3IZ$Zh>V9{@8btTFR=3(n1Qjrdt*9sGZEF z+RQ<#EI;Yl>2z{xK;bO59Z9Y=<*n6x_=-^8=5DJ`4SzbK0}%=2q^*WG)~}7xUsXo|XwBd6&tq&Gfa^ zCnNr4O1?F#4{4YlrcW}K_90;*4ejsp$UZ67`cK@xKe$SjE5SOuhnr8#hTeS=cknqy zxW(AZ;3tHXffzqL)6W$e3v|B;T}sS?jGVdil7_RObaft)Vuu7w zst2o*$4y+L)MtydO(u*{!B+VR)T3F70ED7}T8Fmz5$iOZ=;upunA6_TELf;S&AWP7 zKuKRO7;77M{}!BLz0q29%;Jb6-CBvt03eERwjrh2-iQ_7&oMN=xb^ zx~1y>ZVy21rtv0)wluK&?Vng&r%k*sVXIE-6@@d~SyrHT2>6&HJw9=P&r0#S7 z86^-$f&dnwgS^o4JoY5DvYH zKhz6sIRyR2ga(E__$Js@iRRn9;S!@uw-1SEUyUXn(QJ50P%WZnooc5Gc;2rx$Q@LR ziI&>7%m?_#QSOQ>jyZ{3k4~b|#S|di;oX|8l>YmMoXHJQ4BQWN9wkg_lS1fIq(+-& zxbAwURmslSv+J5}XP-Z{xJI7Ju%*VXvARR90yI=ki%#>bBX-R@A%sp^E{1^^jRg`# z7U#Tr@0^XwQm8bw<~1p?`A-6;Q?mRBw-hSqVt-ANgh;S++~C?Z2y{Y!^C7w=on4tp z(A=FmSR0{VEih>J1S%#aMxhJ|KK9JXQov5|HC!?(j)vX!f+uLU)NrmVJKG3*Z%#*b zu6SY6$JVSx3;@gblx(8JkFAIIkLsUW-tA(Q5KiA-2hi7e;1?h#yTH%#UMH4Un{-1sEOng5gh9Y+`}XJ+SBVZU+P zFB6XP15EWvSDd0NtyAQK;tb)slz|Ql_n;;G zwvyG zCF?JR8Plp;DeXEyZEkhu4*2J1aT#~T&Vq`)9ISiJTBKfBct5G`?d+8J?xJzMJ{gyC zTY_!X-t>v&T5;k|97<#ha~Al?;TE@8@7;GccEPd^Q%(N)PWZ)g3jzE1rO-S?5ZCXf z5vDnQo%ZYQy0-U?JJmyiXN|z=>k(}mjNSKWK<@$V%F=A+jeKs6Af{I7U2v?FZVroL ztP-5&VaQaO^RmEJH3t=2BxGcY9!5RcTd%Nfb>ubFxA1U|nT;ZRw9GE`44Hp$*c|Wf z&9Nv!B`{}fg0C87iz82dn@Wc6HU(tKmegGx+qh&%s;UGNq}V-UZ-fEx7+39ho5;*d z>D$T;yVsIuVB@DLna$H0ag%7Ag1}kJk^N14n}x?Xkh~h^?!bpQJ|lZ%v44lO;k{W8 z-#9e3L9Y1N8t|uu!1PoSZpj5roQ_LhN~U&J+j5B**k*ITpR0 z_14&j3xHhJ*O3j8O|*u4n7{V2W{ph*pCIBmxzCi9L6SF3q>M2?PAY=o3mr!@Ba>eFA+gg|-$Uo$>#M$( zzE8rqUyFC=mwcm;2VWdzrEh`AEimi3#NSZjb}Ms=a2LMinrejuS!X z9FY>y5my0Lcw&V=1_O3+8#WO*D2Zv&JBk$M$^h9QoKj~b+kMiop$-`{JV)p-2c79- zp^=TX1&hM0Q>Gj*rQs_7B*k4ZRaJl(N0y2775=)mU@Lo(h(_xiY-#g6|MT>q>e7tP zl9MrNZCUQ=gT2x)ohqqQ`?RqJU3YfA>S#?kI;sjIx4*epqB${Sz0nT#M$U~5OI|!a zD!SUP!`4I1Tf)I7QO#lPC+OS2d6UR3N&=1wllTi@DS0PwL5tS+rHf&c3)_ILrwJR) zYCN7D1oo{Umn28J*!NQ(*A5tQIlpizSKO2ji_Wmo{=K%x2VUdky2$qPinPkyo&L&`IcTnK&hGv(a{Q2NqyZicgWYU>Z7 z(sOEs*%rsK$4(tAjvRr>c8XeIdhz6PRvKPNqFG8rul+3AdPQGufju*WPTL?ZC4g1l zkYoBD1F!_QCgLJZH$?-BOSzpcKWN7thq48gD5G_D;Fp_?O#^%a>b3=<<*j1u)Kv*I z9X4)-teVn)sLPHW#6}zigMtISdMLb14vW3fQ2=NTGnX8Y>k~v)>5>dgV~% z#)k2UG`fw2cSS|y*D3I^bAn3fB3-;l+i+(ZAay74(jZTldRt1^+zM)nt`12pU6$GQ z8ZYc2aX02kse9#r=bxK5u6r}w!m4uE>@A}#HwfFKC{Vn{DJC-27krG|Cf8D9PqC*S zU<7B}vxM|2nUXyA<(D*6fpXD&HynyN;S^K05gxo`X2f|*=1xD$h=Q1Pi_=Qm(v);m zE10u-yk-jRuEas`zD8#>CDsEZ*xCoa-jk zvcgu!-|;IMdNE>>Kl}L`H|)8GLk#`o=v$wNlKicizU^NaURRl%@Df)<3mC74aVXBB zrGrn&qwSJp{6vuf8*RKo_LPkzwj@iaS!*Q_Y3ZQNQ#VUUV!0MjYD3EO%;NN$?VwA3 zf-WBR^ZX!h1T(_7fp&3yl2$t1rO49!0s&hI7&!>NHTH-xz38HuwSFo@)c8JO^_p^U zAFfYU!+v?h@!IJf4+ab~9We9Mgcq@R-rAunCqQ492>W8F`lmA0z*Aq)j8@GK+t0~h z;MNJ6-J}O;pZD~LLKWbF?g^uuJK67<~{6GhcmII5n*PlJ-A9I!pNW zij@6zGya|P3N8cw5W~oHIW*zWt2xE_uMI-#jl1;PS#UYJGrUI-9%2#xkOYA>=TLaN zi8m14mdE1$1KfP_Hi-K<%T&HGwo`T})Ik_U!f7MD@4eNqBH!s*4mH za%NW@-Kn|7VKu~?f~8p$PJ+~Pzt(Zoc?7I2hs9$NqwH-5Q^KN;%ZhkIs`V6tc3 zz7?QZ0LWRDX+>9zl;4iiyQjeGUlQPCJTxpz5 z(e1dZJRZK1g3+c+cIZkPZ@Rv@^xt=q)8A4=&s2I#Bt@7WE7-evL(q|*rx<}duukyx zZV?^aSoKDu%lyhLbr8kMHY;MiqF{Z|^Pg&u1PVTgscXni!Lgfvh|NRf3`;aprP?K$ z49%im&vNs8C$~xaMsZP3gt2rv}>Gi5!p1dH|d|iZPd@yYPRVD(|*6 zQ!+H68swUmSW=t!D<*rVF&>RzTw`~eAsDhA`ZrN)nW)c|rSg}^%&xiRUGkYhpR()7 z^F67DF|kz>tH>hm?yG;#iFOG_#J+@w3i81H0e*hYzJe&fk7i(Kt!1u|qG*W-OzW}T z7JUxSTy%M-I(3Uu*&yLdo27o+Siu3hNu`P{ z>5+d!_Rt(X^gE$eUD--DcW0P78rQGWkTw-lx}md`!YEAR9qM-v`u$Qvf`5O z5#MI|rCsmo${yq{ur?))=6%{X_RDSIYiXX$G(kWt*4c-idV;?t z%F$gScmMAcrENK~3KjPWn|jiTVNX20Xo*)jvl+t>i=2s^a{kZlBe3s1viU(P0__taoa9*hmtuw=(Knm;VRBH;*UC? zWYtByah3wDF#pX?xTRuJfJz5lrA2k=x>t{ql=%2)fyy*FI~c#~O=ibwvoVoVf2n8Gfp#?a`n zHOb(H$lS`X zPCq{tUE9R!FVw*F(V^c$N4XXx00TB%4u0vk=qI`sV!nmFSKBwP;g{ zkokMUOg2OsuK`!lG#JS18)dy(=ShJ-_wE`yhFs~cVV+}9P!P|UUP{a+BWBVdpIyYQ zVzdaofx0w^OgJPMGd7a`CwFQ9h4x2Zib%Z||8O^B7@@^Y^i`x(Pt^#tVYeyvD8IWS zQO721=>3O~GY)u_^_`rb)}P#{a;y}9k;O#T(NKvnHb~vfnk(4Jqu@d$5qc+4P*1nc zJB^f}uMbhK?mVVaoo(-}?zeI1J4Kgts`s1fM&ZB-u%WB9K#K?+MI%@dBEs378jp#R z>JlqS8|CoAr#8(}t@p~_uM%KWOgLski+HE?sYF&A@$V1PwLO6O05%rs%dB`+Zi)20 z@vWKWbi7d)*~4Kzi*d)aI)d!Syr=psR2l}E(izCYm9DH!L5C7`EtN(kSDqK{8Dt|v zUxmIm=k^YZ>gnm>_=qVj)bOFcL4>Y7tDy-(J=O_(KFKBeQk_eNBU>}~f*j9f*bj>S zWo|1scqUG`MzaM0)m(n5M2^y@M0|$d8?$EiNTur7f6I}1)G@7fQat*z?}xf0UN*_& zD$l?b4l$tQ{g;oVQEg%^cjUx2-LjT|sS^7J%YC4H(16BTs*?PqSOc7{SlJELMeRZ^ zJAYppo+dLP(n!k1J!G-m{N0-_R3cXc=tunmbz5$6=Qc)_Md$nK=Ei2Cc653qIt$nl zvjxM#(M8u>c?+;s>!GllSAjRVYHQ~nWKvmPmPFTKB6aM-u%qa4{XtzmQ zSz~pt)cPo)rXZ*^cA1wxp52Q6IP-x2LTW_ta-}gGgyeInDwZxhTXoS!^o`?~ zHj+u-sFI5qK=9p+{B~@})A#da{k;^rAm1Lr&hb?3YtIWLWj@b1=vZ8}2D&jOqR(6{ z3a)%<_epc+nc^TFRBSZv?(Rk9i7Vtf@K@7LQQzBsb+GEBuXR3Z`Q+bq{_gJb4=D^* z%ZI(21Sqi5=Pf7Y=*OMr6a_n0d33oT+>lWsRkf?eJUzjy$BkmY`c!8V0q?bQU9q0H*FtLuwaQ(wARqvhs8$X8L)r+L=?vPSF}c+sa{$c%8|=PqBGh z>^FbewcGSbcm5%2djOd;S+WS(6glS8a@*cN!yV-0c@onbHy?a!@=1R$;la=L>2eA6 zBk$X!XGlLVvR((3wl~xLn}P#o6?}20bc8UUePxw>e#OP1!Y83sY*`mSmpzISo&HzX zZXzNPD~3|}K>57ZAW87CJu+1YEW!0UZb_jb$xb(HZP=OET5msjo&SBj_-BVGalUFL z*iR=o}4GVzQFogQLp+rDjYr5Visf-{WRTD3?r)P zPkiO4f;q3aUNA>NFv z!TI{cNI0L~#^sA8F`j})ihAr2%dgD^)ow*GD&#nE;d2c@^m+yl zf%+YMVV%8z#XR3Gy3NC}tHDJs%s5!Sd-S|D8yf`4%!>|g!B{WFk%6@#ai&O!H=*Mj<~2eh@ka}l9^^jegT5j~GZscAZBM!; znKDPc-^58q4Tp7Zep_Qo%yA4HgB z=!+M=^W(|liB6{+z{dP|bc-cC^_Nyp3Ifi0b`v)(la7!6zV=BJPN~rCq&j__s_@;x z=9zIb|H=yyqSF)`=R;_(ocY6BpegUcsmu)gw)9v0e)n=4TeB!?3-j>#CrS9;V#=_9 zLMP*?nH?UbJ`>8z0$)xF*=Za}iyx@Oy7*V6w)z$69CLy!!j+I@MPXrBSv>vtx;@ed$ z`?>1b8FLrxhP7VH$7U&@zuRehZDNMK8fjvdf;jfOg~g{Blj188(*ZcEc1TuQE;mYH z@?c!fN#$CP1orNYt_Cl0OlzH+pN>d`PeX+gO0?uT`yBHItk2}_6a-=+l{fOKS%7{@ zQqX_&y+qBM{RGh1WurvOT&yVc66~R8&e9+d;sxB~Hj~r{)9nz`n%kVRmXM5Z9geDK znHGE^L0L2{h79&5f4K#MAlr;LapDfslfT}C{|esUQjNljsGqAfdTdCTVE^?uJ(j6r zZwGn~Gk3PCe*EVfJ34=k_D>I&iTKl&YOH#xcvuqATR3O0@j?oD7*e#E)58EM;)x&D zqQ5P`+C|HuZXani6!%>N1qhewinV=X-05b!2z&RfksvwXn=ekDp^wSG&60oXEhn?w zK5}<@fRC1?ZOix2JN!22+=V$adu1;554EoQ zeD&Uo6uIWW()Y2B4OPAuZ`3Ntuxpr-(Rz<%r%gH?5x;O|9L;6oD3iZk=?_qy$(zZF=&)5^0(o$h zGq=)MCMAxKaNTLBoP7QEsHFOHf-Eil=H{-0uqwNL)Q_E<_=~<7MAn1_0NQYQv3WS` z77aT4N}W!hyXBFWPC{=0@<7t-e}l|vpxobsF5jOP31R}!Mc!Vott(Gz1#6m%xQqG} zvs4;id&O+f;z=*H_{2-%VaGLMva2JUJl)oa&yTR}AUe781;raG(E6&iOE6<1LS+}>wh4%X27KQcavKx-Z;`AHY zANg({Tw96bJ{&Q0_11()zUF9lfyFM3lYJ5kzy8=CHo?bN{rX`II!oWVu+m5<+ZddRs>W?=+hx;it^5-wIut5 zW`+6`B7(24C;9iWJB0NKi|~@NC);P@>Gq=*9cZd>1k1CYnqWbv;T9qH*$R9h(wc5G zu-e!{d}5~*PnQVd(kEIl;CyigH?J3YPdCCneCzC4^Ik@UG*|X7#^`x<2{N=sDx$hB zt&CLL4ev4ZsS?KJ%Lrr5m~*aD#ePs3 zZ*){czT&!Y=QDPx&oQi@I`?CDve^&>h|Q5bcgq;GXMZ}T*@cS|#fRTCGa98KdTFm1p<4=gulS4afBrbXj0|}Dm=Gk=Azi$5O)?vD z@jB~P)$lHsE2x-zAV@(39?Oo<=PvLSU)0Yn;YZIo05Kfdn}^q{xKGC{HihPK6*hDw zw82a-*ezH|x6;w(38{9!)MufSaIYDEeY>1+^7h~b(a3&*7}+2@sLk$9LLr9{)+T~D z2&^*=2z5FX{8$o*&>rEk4dyr;yEQnKlk#UV_sPL*5_>-~BTv>uMWs@oxvsgZQX~$0 z({bzPu^oNUkgW=HMw-fNRCA$&&tEadf`)G0)Cn&`H8J8Jh7+FkS9{fKbJM zb4gpNx5dlk1#i}v;au3iT^u*I78ZSoVit9|J)Tn(*MEfv_W1$+!*39X*P*d0{&(q= zM4{0ZN~MZ;LzI=Zw9N)@YE>xve``ncq>tiRZnxY19{D9B*R)P$wT%ncOCOXYf|MGkJ$CZ& z#Pf$WpcVLLY$E(2+n~noT+cPg3xxbSV!MK%%I(gR(YCMVRqYOWI{9c>@iyFA4lDOH zaaV~Fl}h1u*QjC#b?(&9eN*wNT75kYQ~VwtqlKCq=9bG*fYoQ~Y~*DEz~pX2%(;q7=8N(ZJ~r?|G5p7#QL)l;#@rQ+-CdKxEr zt0}7D;eM_9UV)!Sk{9WxtXen%)y{iDu$Lw4?%Ipn1A4JzT4BCO~ zz>nzvO_d6!Q4YOtz`8JqD)$xrYWy2`s{KjOR>15ng}zQQ-le`UAr{Qjp;B1?;MoZQ zN@*}6llXQ16Tgi}1j`*#g2B~GMEmObCWx3h7r5K~2e?ZRP++iPXZs+}FBF;r?WbV8 z&fn7&F5k>YgE-u)q@^lQWZTUda^kYRvOuM%jL951*eyy+Hb#|$!!r35FP-3ZI?da1 z#=OsV1(xI2e}HTn%sMXs10#z3==Ysmk**ope7OoVu`)(?J|I*h$KX z9P5UN8cLp8?Adru*$ahPF^AuNL1osZ(ooNdg9H{Lcroc`26$}dBL>V(*T1B@2-AcY zhGI=L8Y^qT2Xemfr^Rm;xbjF}=R{iy#Hd6K&=`pg|&3IBaQqD5fP;~uDY zUZ7Q7T5YuHsNzoGvD6pzdl&OSGwX&_Kj&1_@{+jA!>E6A~SY5Im1;C^uwK}wviq_%Gr1S{BY>ovum%8 z--1!K4uU}T_L*aMc50TyzXx5FDmkZDU87a!f09J7xMMNighDwa4L{9vxl9ARKeHi! z(Q#K7V{(Ps?b1Et=?@mJU}1RDDqy+0I9mXxDW zH`5sx;W7;=k!K|sWJ^vdEZ7~-;3Q2Nz6 zb=1N{5sk zX9?b_MgeJFkg!TS34Z8{5cK&RTi&<3%`$XPjGdBanllAe0BYAd*iiMLS@Z!k;y1C7((U4=2(m=WzJIa`|Khfpy z_uA>L;capG<%r6p3AQ`q-G8{aB@q<-E2Vmte!Nzbf13SiqH7_P1!KZf12}#4rLWhE z6eu4H`bLL87li%?@cXPm>4`tz_~Ih5uoWt|E2Q+u#!!~~H&P*JlM3b&Tai2EHS4}p z0qF34C_Bfx^cnJI zI06U?Ph9#QFsrRmnfx#4n{uU_>Ye7hWp-7gf{wg*dfB!~mO~UAp9CI6tW~)4T*&*tjz@0 z$7vc7+&+SC3^12nIy}DtwxG-3Ggy-l>tr$v;zSEVjw==u^wU`38s7Up4^eGUL-Tvh zH)sI-()h>rM|5>O?+r%OSHGu|h5nGsqJoUX1WVFD4(^s>L51Waa*d$h=&VIkDagbV zxX;IL_uAPFKP;+j#qcR0>&v|ejdwM7ly7ppP|4Y(+Hay*W4G2q0a={}`m6+?S(FqjKzBoe$vne@gC{coH63A+r zWTA?KF!z_%T(XlROXR8|D*X4tIzFXj8p&$VB=7p~1@m;u0)_}#pWUTf)t(81%#Yu# zSQ%Sbh^=q3s7|E@*3DQFZ>!5s5`}C2DA5aC6D=PT>M8uGS7h->h{O>AAS-{InYTJ^ zYnOn=Trbj2LE=0f1>0t3Z~4*glv)m{Hsom1oj0UIgSbD}laz-=`g!d}cj?zuRluZ2 zpNYw$Kry2nj;-Q0V4!oBKMdp~-i!Iy|4Uz=R}xnr_t0S|DLg3Q$QepomZ!_@k1g?O z$?BF~7oB3^EAaQns~{}<61h8N#mrF~djSspDK95$&6Vhx@_OCeeARP}L@N}r#S#)c zm0Z?6Nk#Wc?YjAdX+FBTNG@Tii$fQD(&J8!^`tUWxqzl~E2Kq=_AxgTI+<7_$Kos2 z8?RBfM~nK*tD}X##<|wsMsdY;=m9PUhtPHRT-k^#BNFydX!{@w^SSux0YLSZAWucF ziM95Cv02<1?M+hLmr_?%#gm0MBpw&5M9HkgnlC8+hV6#3{{HVhbvX)mYLZ2#UvM5t zQiya(qld!ewSmUoqC#3}>?mDNy2QT8j^QtOkSuu8a%m@b8yaUY@+fx6d{{^Js$w@M zwjIJI^ogO#O|m%t^?!i(wV@N^+(Jn${fIk=k%r@L~h$nn=(Dg0F<{V>*bQ} z)C~?co+Z^kZ`6lV>$4GA1g&U*_#blrg|3*;k9yX=QFS*h1#2mMDkq`b{(zCLXVE;5 z$Kt>teY=KD(m9TYK=TI2X&Rgn-$D}OB+m?rlECw2dFr#CZukPH^iRSnI6z3L`5l2k zvNVcKd%(oPqmau1FOu}$YV%uJ=0Be#C&0{ve%#P{*3W5gdgjdL0*Y}FU$GCXxU6vgvMAXeQ!H9_L!$M{$qI}jxzau zh~LV|L6mnkq}^XIsuzUw@$qjQuxx5d@reodOe0P?@s)or8k1=nNX8AYu@Qyq6`_Tq z-C&iYmA^3s#zwkRE2VjcI%`>?43i1jiY0Pq;Zh~+rK*&^+Ad<1g2={dO-irirCUwKt--Nmb3$p3{9G^V&nyY9%vOh^C<4Y z+RAi1%2Q%Up%0{bDr76LfT6jEs1%J;2(zXWu7n#0-4uG$aC=l#?5ea8(OC?&5vM&l zp0r?=P2iqrREw%NZfXPm&6>gWk7g_I{s*W5zX7Lwb&gHAE;eYX@b`ASXF0+n6S^|f zT{|GJ8c~?}=2^ThlqHOntD{pMaZDOvvhLS7PZ_7dzEgCiGghNpwfzS#HkPS#IxsBp z-B`#F7c0r3m(LrUIFa<6peyGee&ml#$mI1ik>tgKk`&%ku6y>ofyAPyWN`NxeYH$OJ-)UBBQKg7cpG^_XR=1E$rU|+!nK9yO0HDr>i9Ho@maS+C zqf=7QRVJsvX1K+9o6+p(!+sqOY*oZdlw}i37-#BFq=k)9N^rGUw%c7Ambk-Q47=sM z(o6G&O=s=TpB}Z?47P!OZmlctJc*Xg%hMe?M>Jvx#uNb2rw)j{!U@7>#+8H^KIc|4 z6ZG}ZPSFhKF@8>3{JQWTDFj=W1mhPkpJ!fxL;A$V3Q`0^-KN_K^aZyVXI_qFx={s} zLtk!gF^$+&eF1{NAHh`r14wz9+%Wm@II<3F$U4XCdtwWt!#LydgKDStfGSSZ_-X#- z@~0%at?Y|TlMH*z#vx0@_Gh6qEP_+GFt)f7qOTGyp=sIOHpdEcUzF2-8Egs%Qx9D> zslH)Q$n@2zZ+f7bb7pYt&_v>?nf8|>@F>TxF8@_6{~}CR5h|x+&A}!9k_1|<3FGuU ztbEpi)628$o)@tr;8dMb89Cf<-FQBGUzDEZNwTTae9$4^b%kv?YA%EvD7xp)482yC zLzJJQh*8rE3MTci!Nad4cpnK&iHUnHwtBh`XY;SR6!g)jE6zP>vd?+W8CW$Dk8BlH zCEpfs@9hADsOFto(5FM3uTH*d#Bw0WzgtT&QcU$&^TOxXilIKYmdgz1GgT_SstYmd zeFS6y@s;!q>iv^sJHTePv`(|E;`3435%A))a###yEr-T9k)yEFTDo5`W~&Q8+Lw*i@H5 zA$+Q=95TSo5)@1;x_0X%CozcXOYvR3N3pZoFkKExBj3cmr+Y_?ErKHIRy5UW!;JjL zb7D(<*Ind%)6-uGMbwC{98P$hsHsWRb@p&9ngZ_`**_fto}&+DP#V zwnJ^~696zPE6H|QZg+IWd*dqfQP5*lb5Q4h3{Hs&c3mF<2k*u&U|b@887D{)RRCeI zgWtsk$opTVzkp9z2Zvmle~NlZ$U}bwrN2V1wkC>aAN7XYWuGC6@S|B;j%b9SKdtp03L^GoT_oQ1sGZ1TEqt&OfRO^FvZt)~0+1J#v;S?^0J*0dOCi;tBo z@aWTkyb>ahJp8#Cr7L%K>2!Pk)cOZAiBa|)i1t@q0_FcMj_`PP<%|_s2x@JI7U?Qn zNK070pmqf5X2bt<^33gf{>GxLUfu3rqogJ{Rp~@Awsw5wu%C3N%5Y!+I zc$hG2gw&|Fsy#LL^8;0#iw=c{ip3B9d+kM{V&KU5_8T>8E6cc7U!t18uVbS7LXhs> zwx0@(-27y0>f9nuil_|+{dlZ^q?Hw@TG!Z}K$_;#Tb>1h@M6wihaa%auQ6=KR|ih2 z{j?j4$KdxPVjeYNJtnQzqfNaXC>#s-p&1S$7$m^MLT4o?iTLTqZph~*$0yt;mVD~? zUm@l3+00y8T41Ckz2#4}C+nhf&{I$J&6cr}_j^2APzkKdyGR9hFyO98I)L7tkO#WV zdx=O+CF1x}!`WWE+2br3D41d&b9=Jt)$&FKGZ4|C5_ul|)=EiF$G8IP1{bXCy;URWMD?dqOm9 zCqOpqN`1@={k2INEp%E^LKlzv%BxwCh?MqNDF$t8Q{y~}W;m>RI5^HpB_`(alN zL!b)dn64bbU6@KprD?~)2H>R0Z_S>WJ{nk#bn6r`7Sm(@nXZZJ^rldhXBK^{Xr&Fn zvU`7@;R-5X5@Ne}25Dp-547S&&2PEVrnM~`C_2yCLGrW{9D%LTAEWY0$2h;+UeSiI4F&!OaI06h z;3LcciX{r2*Lwtc8Rc|PF$y=^G7owMCJ2)>9ED@Xc4I?~dp^X8pl-YTQk2_4nu6T8$ z_6FWIWlCOUzTL^>s{^;RdV6MUbkk08U;WoI|A>wJ#t_( zGW!$peZq;SH%?YCJ&Av7iZ?u1-UNUXQyX*#Ykpa=*+@?bhMc$FMwN=|MwS*pbf~K3 zwouG@UG+fh zl!p;ep9F|qK#vnLhPIy5wiQ0+_JO%wFR7;co13`$7;N8&NpG{_C7S}WlXpu9*N2(w z&4t#BA0q!AAK$hRt@}{{(mU2KUJSAow|KLs*O_!4Q^=uY&GL6GU9X7vy2uZpLqY%b zRvRIPE}I;>$PX#zD9N2<<7AIp88gZe5f>}mP`ao2RdVpglQs`<_S7dKo}E;3n-prx zxD{YWDduMn9k9M%CWSsT*}ak!o&F%~Ir+<)Y-HaBoD#-_l+Xem_8a<1UNKrLb@!S^ zT|+43g$0k;i`60UnAtiYUPG8SLqg*E!RKSciLtAt9M7|5-%a1lh_wmk8!Ml!0UBC- z_OM@qP%~H0mK)8cEDF!?|Bc$x4F8cofcJ(mr6_K}0r1f9m z$VV03vl0t1=@^w-xIQewgoMP8ZG0 zm+lT{)1hRtcx75;oVrsarpgLSBR~B3vCK{?cyn}Ebk?%1VwwjjUih=vCSCiLQ z%uc+ymP(GZ)51}xQwSK7CHRG_8F*3kKLNK8NbeJ6gii<|@XzIF&k^MMTz)EQR7}9~ zN7mx=$pTgwQNxXD&x$yy#3S3x*I7X`G$0t&c|d_74^5p)LU0`Mtrad72+_yx;T9M+ zXJ5r4%Yy+Vzmo3aCrU7i<+vYMbxzllVLGD zt6Y+iIf!ggIYbM!+}Xf5PEHnnXq|yDiPP%7mn4`@9^xi!%rOA~fr*&I)hTj;B4P)6 zeAr_I`?I7f&^i;~+*gJ5*eNcdm28SX$!*pdx zn^;Wu{{U)Ij1+*<5y=%lV(k!iY-?F<_A!%IK{_)#{{Xa37wnwHk!I8|4-2Yk5d!V^ zNh&izi0ZhZ+d(_{pfXG+P-?jV&S6xcNtlqH4sL8oBNF#;3Ze+JSTdwC4@$WjfIws2 zaw>wB7yC7PSR`c{vq&9^i*8y>@Wctf9%~f?XoSx8@c?aXeoGG$VU%V&)ky-uAR<>T zVC!|r0|BC8GqT4fiII_v3~24ViRMmZ2=;J~9=!3L}xo?3{5WLfFC->2tCfr#7ZT(su}6 zQm9JoV<9%jR>+!lmT0F1ra<(gH9`q$Q3Q!x2Djpfp;cg@OU+B=1p#q~wTj{6zYKD_ z-Zwl`keEEqab8+RVJn&uz8)^ugS@jd@V|>bHb)wA!I6Q!?tjyFBM<~pm z_^A#SAj#E&IuSAOs2Jo?K~%U*a%+NJP{~Ng4b-M{E47~ZpcIBV8x~&x4tpE|n1`;>!xeV0~LNS`DB3U!I$L#r_l5z-wI(%_+r9x99)2NzKN1#YZ-2VWU zS}BaPU?!s?osEGAeY>M^)p9Ij)P}5>afHqTlAU0ZVLA&b5=_7v&6~Y~Pk8>9Yz!))tGOpamUTK?gDJcdKr#YGk!kL|n zqr79i(2zu$G?r&9k=SmTIVn6*=;_qZA((Jxki$$2Q z2T80L`{K+YAV}gLhVxEFe0W3$Kd8+iW1NBoRlp)*oeV>Qo*hr`nZWU6Ku6|)MnnSv z?yf00CFa8;bfJN^Rh6Ki4gmNeWr7M)WS;2B0gX5Cc z(I?g{*AfB|f4i?LB-uW6M+kBe=gx6J`0}0?RoH=nF$!}b9$&3gnB_o{pNc)a0iP)5 zxqU%6d4$dyTq%($oD;2-1`EN!`}&%A8W<3WBhEhSb7FABhF8z=TFG%39t6}53(Ay) zc=KS<0d7dtAI)T3bW$bPJ1WT*4VFzF=QTnYutSOQS&$%4C1n|zR+_<%25{jRCF-nY z@}5RzBNDGJT1-k4j>M)I&x3`UGjL{$Z*+#|I3`+}GJ!!cl?{K4lo-j#SrkAWW$)<- z%;O^x9{y+m17$Ke^|em_Qkb-rC2S1sjSf1irns zRFMfwA`Xc#r(+1k8-GxASUNdW+yX8qj>9x$*nn^yLP9g~jDd5&Fht5|V-JosVpz#2 zl_4t5;lO;Q69fQ^rWzdZ&L{$8XyC$3$4aXsTaW;jyh07cCUw)AM8$zZIDwr%CMbiK zAZYUrW*xDHm*sx%j-=VHZl$vM36Sj%l^dz2Um5-No65{ycFRi z0x6U*GZ`)>1C^q&RgDP?f|x^%;))p3iX}K)p@VO5kYJIzV>iZn$<@Wzj ztOSp4Q|qc~3gRw3(F7u{iiH(`1ODcr>CWyy?JRj9#Y8vrLWvLtcT{FI)ewAzO`Ym5 z2;McjSY|pv@4KBp2fDES5*69W$K$9ss$8q;gJ5XS; zI^3^{)T&FAG0Wf8bvuTj83&1u^X}mcfMu&8oPBb2V1(g>^DYrOj670N36uo9zSyO+ z*epk;em7L>$&+)K#xc;=TND@oA%A$YQgAWI<*~+e1L;+q0B+`ap06?$1w*u+~TT- z5y8wT-d0X?W(!u6fShxa#x5EPl>)8|%;1}YJ&yA|1mhiiM2Ue3Ie-X)4r5r|#Bc%* z7(nUr+Ho2%Nk(iUC>JDH)GGdc!i zs&ZvB5YERBiYX90UB9~;s0jxUcUT}HkSuG(SCo$eIhyWq7z4m2CMa9oFrD;syOlIS zY|EW(&s4|&LRg7ef4-f5=qIhvIcp%@8h@Kbm^N z#)kxt7;CtL4s-mE#b0~X5(~8oTn3@3hNNd+4Nq|2Z}v8ARy`%wV3A|ZZH=f zu$erechwUzh;EKUX>9%yARQ7}#=fiw=YkLr5Qz*f^Iq5pqLPiUBq!CJKnxPWIfPGC zHqhfdgs$OVL#n3jh^mz#GEBi3GrBDV0iDM(LS+P4iqFIZ06VTt0EK%o+Fd!P2VsF9G94eYFP7GzQa05P9)iPK`hcxV9ilLYujQLtf< zhC*_ohfxW!nVSh#d?Z zgYy`xrf^WwWb;t?#4)D>+uy2+!#D}S&nOmok+KI2eCjB%QW8KQkAZB?ImnzKeXbw6 z!V#t*0WtPJ7GH33RI&`gcf`<4U?adIZQH@s*xZYdf_9IBTsD*sooWKG!jUayJyj~4 zN@7#|#Wl*4A~<@O&xL0}uM?cbaY#g1OJ>+)_2TD}X}~ENmKYb#Y2R{ES`@Pign6c7 zWJsl?A5SQq56Tp*JMd_gfaoZ>@4`On^IGGh}&cq{*e2BLn&Sy8Xj4 z94NO7j3JQxQ7Ix$HkkDk0Z2rG4p-;#OC2G=AScZNCT8a z72$%+=ixqU4yGY8IVK_#j%YzhiZNjs&h^E~11<^XPZisjFOVBLTpq0QUW+PRt;>Vz zs>MvM*g(dtuZr+g&zBnW=Be2MiP;V@2`y%9hB@#Y7@-9RwSlC@f7>kJWEky&J--zw zg0SR}B>0{hc4}oKN=Pb+-sy&kpi0B8Wg|UUKo;DEm6(mj@e^K137=PlNgq7f#?8sX z;U*Ea`>WM`urLDz%7*cZ-h+iAB>74q;Xw&0P{AXC2}fG7HZ>Ok;T9NBoiQ>UUX7Fi z43trXaK^7Qo5+j?T!TJ})-<(9%roKB-HwP62?OZ{>}--LXJ&aZ!H0^v129;dhF2qa zY@E*$7Dgf#1Y>I{;mwkSHyFYr2Q5Rz0|~OQf?yUL4DQL!0Vr%nWq?db&=|y-6wc&m z?4>9*)KHVx@mWiR_R9xQ`o&dc2@C~5=OOiR0u2zJMUmPy`mjaRwF+d8(>sh_2H-<5 zhI5GLqA8iv&P)5>wWyI55H_2YRm~s-ADf0!&k-gG^bvphi7rv+y0vOG!k)pqv<^M+8s|%VDt&p(t@$YvT~mh{Cc2 zOd6c8M=F_({%S%-i3y5lfr{q|Kfzs`!C=U_?uo?Q6H|9n2g#I2;=ee$9L4)JXTdy# z3Hi-Cq9k{jibRtv!7G!^VMqjRW;5AjS%OAXX3twrGlrFfRM1%tQS^ONB$60bKe#Eb zM3t0G$9Sy;lVPz8bz|r*DBtm8-pbx355Z=JFoqb&eMQicm4izI8p~Q@7(fY!CGyXq zW&Z$dz9@wz1Z%4-!D!1kJ}z=iUP&_vb=nm<0wDAANO^_KBmUW_GYLQ>CZa0LUg_Me1Wc literal 0 HcmV?d00001 diff --git a/test/smoke_test/assets/rgb_pytorch.jpg b/test/smoke_test/assets/rgb_pytorch.jpg new file mode 100644 index 0000000000000000000000000000000000000000..d49e658b94ffb204533dc228d8a2a2f6b100637d GIT binary patch literal 2126 zcmbW1dpy+X9>;%kVJ@a5qh>H}BPK-|w^3-eO_P5u!IZj zu(zSw03Z+mfTRPEi~`31xU4KpRt64(!4L>I5{2G|l9NMWcPh%GamwmyIAv8;4Q&Hm z4XyonRaJu7-u(v+jg5`fbA~dKnMf^C5OUn+lC=(s%jGdXGtCd3UI(1 z@P>f20I&iGq5zUS1=OYYlmY!F;NJp)Ay64v7#x9=lLl&c0ALUV0)|3lWS~%KI!d|^ zKow+mY8qO}D$+b*T9?tp*wg|z-nyzy$+=gkZR8aYhd}PyjlnAK(a|OB)iX9RH6xiH zI(D35V{1pXclq|@Df;Q}T+e%d|977Y7a4&;!6Bhx;Sv9cPe^24yPkCOR$6*S=1;e? z3hxx%y;pqyK}mJZ!`er6^$m?rI8UEFZ|``~+4rV@U~p*o?a27Vuai^LA7^HH3yVvi zMawH+#H(9eAOP|^mUR9d>@QpjQZ6tQ3W375xIo}gsUZqb8BIgkomMoM=Ve7LVk{hO zomx=UhQJ#+3zfVAdXc-djmP&aY@z)|_TPcU{a<8%fc=MS1VBMR(&9lB05TxnOd{li z|8PxY@%!06Tim3++zU+V$!pzja=b%k!j23usP`&Zg!(4xcL`|P2`dWv8_=KaaeZ7J zzLDA0UrUex(JKY9tku@#?2SsRQGSn*9e=fJiszaZ-MZHEn6q(q&0%vWe@-;p>zg%I z!SePu*gmVGMX+l7Q36=;BClFEzjWO5YE93!Z>quXLOcI-xqgXLarVY;-lee}5@7Ni zYqNze0jwF5CCjCyXZU7$7n_59AdG}>3h3G~WTPRS7G-(xOx19i5ksPQS5z%WoJj_^lhg5$vNKeEo*p`lI`sfBX`;lmGUqAS~zG2uI7I;GW zmnJ5I_gr;86G(j+VAXy?Xf*Hf%(%A=b*ppKEBS9t=W+;G-ING=UP{G8OHRbko7A}Y z(Wl>z9FCZe&Ju$=@(}tm5i&fgfU%KshvOliO($PySz-!e z#CDjA2vugs?EIJQU!DybH#}t8yO~#qEV6IFTzc3`Lt0H^V1DI&SJUjHW2SwEQ~!kN zt2;(sUSwZkTQj0@7vol}*PKhS5FbgFGXo4?+BVk0k#|>0k-S+Rv!3tow@M5Ds~h_C9lFae0?UMiTWAxdtDCe(`>M6~wwVHJ=lsn}8_G(CgnPZ}I_O%_<11zQ9fGYQW4X>@@T3;oE zpCpedt+Puw5}>2dJ$vMAo>Txlf~@LB->5C+il}au!Sn&EWA|HOw5jz(imc zG8^t=QIml%W~wOhKp0Y`novM=4`VypWd_Q{2N4pSihx$F=IV=_((J|M&<|J82H@qIZ9f&HnAhE<8iQnv%Z=#VAE3P(l=+~Yg*fU zji}zpnb#D#3=b(r3a7c$Y?YZ@#1ThiIxf*BP>d`g%}`w!VEo6VbuKW#>lxoARRl8Bv#BoxMc^&yAd&$Ci)DZ6#<;tj|S zBzI$NNuXAP2JShp#xAn@g%!+haOA1c41?^cXP+c| RJo$4w`r4NMvO*+p{{<4)%02)9 literal 0 HcmV?d00001 diff --git a/test/smoke_test/assets/rgb_pytorch.png b/test/smoke_test/assets/rgb_pytorch.png new file mode 100644 index 0000000000000000000000000000000000000000..c9d08e6c7da91991a780ded69d966fbd0c18eb5a GIT binary patch literal 575 zcmV-F0>J%=P)mlEGZGj7-Nhv#u#IaF~&Ub z`}JAfg#UT3ZorwgrlOmy&ZeT3tmdYokF5TtqKT|6OhpG-yO@e{SsR&(T3LIUib7f2 znTje|JDQ3TSx-%UeE*C;Ugi2tTyNdf{F~uOlZ7=kb3iOQS&ODRAd+PmaMjNnZUjJ9 zlUR+5Lc--_C1A-ayjS3rUX``c4B&cG-3=31RsxEw^1%(M0Zvu|%SvEb2`nptWhJnz z1eTScX5Ns^HuHuz#(W(4^W5BYWUP<8T=~wzvHVw=eSveE- z@vmU*u$W9x_LNA6ouqO*$_do4iT5CcyX{v$JfuD{;{F;oStf}>w4t0GG0jzx?!IRz zQ-)W1Qwl#ZaW_~0ufxGg%Bmjxm&rPqx3c N002ovPDHLkV1i6h2Xz1d literal 0 HcmV?d00001 diff --git a/test/smoke_test/smoke_test.py b/test/smoke_test/smoke_test.py index bae7a5d29d..1a55cfed72 100644 --- a/test/smoke_test/smoke_test.py +++ b/test/smoke_test/smoke_test.py @@ -1,49 +1,243 @@ import os +import re import sys +from pathlib import Path +import argparse import torch -import torchvision -import torchaudio - -def smoke_test_cuda() -> None: - gpu_arch_ver = os.getenv('GPU_ARCH_VER') - gpu_arch_type = os.getenv('GPU_ARCH_TYPE') - is_cuda_system = gpu_arch_type == "cuda" - - if(not torch.cuda.is_available() and is_cuda_system): - print(f"Expected CUDA {gpu_arch_ver}. However CUDA is not loaded.") - sys.exit(1) - if(torch.cuda.is_available()): - if(torch.version.cuda != gpu_arch_ver): - print(f"Wrong CUDA version. Loaded: {torch.version.cuda} Expected: {gpu_arch_ver}") - sys.exit(1) - y=torch.randn([3,5]).cuda() +import platform +import importlib +import subprocess +import torch._dynamo +import torch.nn as nn +import torch.nn.functional as F + +gpu_arch_ver = os.getenv("MATRIX_GPU_ARCH_VERSION") +gpu_arch_type = os.getenv("MATRIX_GPU_ARCH_TYPE") +channel = os.getenv("MATRIX_CHANNEL") +stable_version = os.getenv("MATRIX_STABLE_VERSION") +package_type = os.getenv("MATRIX_PACKAGE_TYPE") + +is_cuda_system = gpu_arch_type == "cuda" +SCRIPT_DIR = Path(__file__).parent +NIGHTLY_ALLOWED_DELTA = 3 + +MODULES = [ + { + "name": "torchvision", + "repo": "https://github.com/pytorch/vision.git", + "smoke_test": "python ./vision/test/smoke_test.py", + "extension": "extension", + }, + { + "name": "torchaudio", + "repo": "https://github.com/pytorch/audio.git", + "smoke_test": "python ./audio/test/smoke_test/smoke_test.py --no-ffmpeg", + "extension": "_extension", + }, +] + +class Net(nn.Module): + def __init__(self): + super(Net, self).__init__() + self.conv1 = nn.Conv2d(1, 32, 3, 1) + self.conv2 = nn.Conv2d(32, 64, 3, 1) + self.fc1 = nn.Linear(9216, 1) + + def forward(self, x): + x = self.conv1(x) + x = self.conv2(x) + x = F.max_pool2d(x, 2) + x = torch.flatten(x, 1) + output = self.fc1(x) + return output + +def check_version(package: str) -> None: + # only makes sense to check nightly package where dates are known + if channel == "nightly": + check_nightly_binaries_date(package) + else: + if not torch.__version__.startswith(stable_version): + raise RuntimeError( + f"Torch version mismatch, expected {stable_version} for channel {channel}. But its {torch.__version__}" + ) + +def check_nightly_binaries_date(package: str) -> None: + from datetime import datetime, timedelta + format_dt = '%Y%m%d' + + torch_str = torch.__version__ + date_t_str = re.findall("dev\d+", torch.__version__) + date_t_delta = datetime.now() - datetime.strptime(date_t_str[0][3:], format_dt) + if date_t_delta.days >= NIGHTLY_ALLOWED_DELTA: + raise RuntimeError( + f"the binaries are from {date_t_str} and are more than {NIGHTLY_ALLOWED_DELTA} days old!" + ) + + if(package == "all"): + for module in MODULES: + imported_module = importlib.import_module(module["name"]) + module_version = imported_module.__version__ + date_m_str = re.findall("dev\d+", module_version) + date_m_delta = datetime.now() - datetime.strptime(date_m_str[0][3:], format_dt) + print(f"Nightly date check for {module['name']} version {module_version}") + if date_m_delta.days > NIGHTLY_ALLOWED_DELTA: + raise RuntimeError( + f"Expected {module['name']} to be less then {NIGHTLY_ALLOWED_DELTA} days. But its {date_m_delta}" + ) + +def test_cuda_runtime_errors_captured() -> None: + cuda_exception_missed=True + try: + print("Testing test_cuda_runtime_errors_captured") + torch._assert_async(torch.tensor(0, device="cuda")) + torch._assert_async(torch.tensor(0 + 0j, device="cuda")) + except RuntimeError as e: + if re.search("CUDA", f"{e}"): + print(f"Caught CUDA exception with success: {e}") + cuda_exception_missed = False + else: + raise e + if(cuda_exception_missed): + raise RuntimeError( f"Expected CUDA RuntimeError but have not received!") + +def smoke_test_cuda(package: str) -> None: + if not torch.cuda.is_available() and is_cuda_system: + raise RuntimeError(f"Expected CUDA {gpu_arch_ver}. However CUDA is not loaded.") + + if(package == 'all' and is_cuda_system): + for module in MODULES: + imported_module = importlib.import_module(module["name"]) + # TBD for vision move extension module to private so it will + # be _extention. + version = "N/A" + if module["extension"] == "extension": + version = imported_module.extension._check_cuda_version() + else: + version = imported_module._extension._check_cuda_version() + print(f"{module['name']} CUDA: {version}") + + if torch.cuda.is_available(): + if torch.version.cuda != gpu_arch_ver: + raise RuntimeError( + f"Wrong CUDA version. Loaded: {torch.version.cuda} Expected: {gpu_arch_ver}" + ) print(f"torch cuda: {torch.version.cuda}") - #todo add cudnn version validation + # todo add cudnn version validation print(f"torch cudnn: {torch.backends.cudnn.version()}") + print(f"cuDNN enabled? {torch.backends.cudnn.enabled}") + + # torch.compile is available only on Linux and python 3.8-3.10 + if (sys.platform == "linux" or sys.platform == "linux2") and sys.version_info < (3, 11, 0): + smoke_test_compile() + + test_cuda_runtime_errors_captured() + + +def smoke_test_conv2d() -> None: + import torch.nn as nn + + print("Testing smoke_test_conv2d") + # With square kernels and equal stride + m = nn.Conv2d(16, 33, 3, stride=2) + # non-square kernels and unequal stride and with padding + m = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2)) + # non-square kernels and unequal stride and with padding and dilation + basic_conv = nn.Conv2d(16, 33, (3, 5), stride=(2, 1), padding=(4, 2), dilation=(3, 1)) + input = torch.randn(20, 16, 50, 100) + output = basic_conv(input) + + if is_cuda_system: + print("Testing smoke_test_conv2d with cuda") + conv = nn.Conv2d(3, 3, 3).cuda() + x = torch.randn(1, 3, 24, 24).cuda() + with torch.cuda.amp.autocast(): + out = conv(x) + + supported_dtypes = [torch.float16, torch.float32, torch.float64] + for dtype in supported_dtypes: + print(f"Testing smoke_test_conv2d with cuda for {dtype}") + conv = basic_conv.to(dtype).cuda() + input = torch.randn(20, 16, 50, 100, device="cuda").type(dtype) + output = conv(input) + +def smoke_test_linalg() -> None: + print("Testing smoke_test_linalg") + A = torch.randn(5, 3) + U, S, Vh = torch.linalg.svd(A, full_matrices=False) + U.shape, S.shape, Vh.shape + torch.dist(A, U @ torch.diag(S) @ Vh) + + U, S, Vh = torch.linalg.svd(A) + U.shape, S.shape, Vh.shape + torch.dist(A, U[:, :3] @ torch.diag(S) @ Vh) + + A = torch.randn(7, 5, 3) + U, S, Vh = torch.linalg.svd(A, full_matrices=False) + torch.dist(A, U @ torch.diag_embed(S) @ Vh) + + if is_cuda_system: + supported_dtypes = [torch.float32, torch.float64] + for dtype in supported_dtypes: + print(f"Testing smoke_test_linalg with cuda for {dtype}") + A = torch.randn(20, 16, 50, 100, device="cuda").type(dtype) + torch.linalg.svd(A) + +def smoke_test_compile() -> None: + supported_dtypes = [torch.float16, torch.float32, torch.float64] + def foo(x: torch.Tensor) -> torch.Tensor: + return torch.sin(x) + torch.cos(x) + for dtype in supported_dtypes: + print(f"Testing smoke_test_compile for {dtype}") + x = torch.rand(3, 3, device="cuda").type(dtype) + x_eager = foo(x) + x_pt2 = torch.compile(foo)(x) + print(torch.allclose(x_eager, x_pt2)) + + # Reset torch dynamo since we are changing mode + torch._dynamo.reset() + dtype = torch.float32 + torch.set_float32_matmul_precision('high') + print(f"Testing smoke_test_compile with mode 'max-autotune' for {dtype}") + x = torch.rand(64, 1, 28, 28, device="cuda").type(torch.float32) + model = Net().to(device="cuda") + x_pt2 = torch.compile(model, mode="max-autotune")(x) + +def smoke_test_modules(): + for module in MODULES: + if module["repo"]: + subprocess.check_output(f"git clone --depth 1 {module['repo']}", stderr=subprocess.STDOUT, shell=True) + try: + output = subprocess.check_output( + module["smoke_test"], stderr=subprocess.STDOUT, shell=True, + universal_newlines=True) + except subprocess.CalledProcessError as exc: + raise RuntimeError( + f"Module {module['name']} FAIL: {exc.returncode} Output: {exc.output}" + ) + else: + print("Output: \n{}\n".format(output)) -def smoke_test_torchvision() -> None: - import torchvision.datasets as dset - import torchvision.transforms - print('Is torchvision useable?', all(x is not None for x in [torch.ops.image.decode_png, torch.ops.torchvision.roi_align])) - -def smoke_test_torchaudio() -> None: - import torchaudio.compliance.kaldi # noqa: F401 - import torchaudio.datasets # noqa: F401 - import torchaudio.functional # noqa: F401 - import torchaudio.models # noqa: F401 - import torchaudio.pipelines # noqa: F401 - import torchaudio.sox_effects # noqa: F401 - import torchaudio.transforms # noqa: F401 - import torchaudio.utils # noqa: F401 def main() -> None: - #todo add torch, torchvision and torchaudio tests + parser = argparse.ArgumentParser() + parser.add_argument( + "--package", + help="Package to include in smoke testing", + type=str, + choices=["all", "torchonly"], + default="all", + ) + options = parser.parse_args() print(f"torch: {torch.__version__}") - print(f"torchvision: {torchvision.__version__}") - print(f"torchaudio: {torchaudio.__version__}") - smoke_test_cuda() - smoke_test_torchvision() - smoke_test_torchaudio() + check_version(options.package) + smoke_test_conv2d() + smoke_test_linalg() + + if options.package == "all": + smoke_test_modules() + + smoke_test_cuda(options.package) + if __name__ == "__main__": main() diff --git a/wheel/build_wheel.sh b/wheel/build_wheel.sh index 08b47335af..26df3d71d6 100755 --- a/wheel/build_wheel.sh +++ b/wheel/build_wheel.sh @@ -97,16 +97,7 @@ fi whl_tmp_dir="${MAC_PACKAGE_WORK_DIR}/dist" mkdir -p "$whl_tmp_dir" -# Python 3.5 build against macOS 10.6, others build against 10.9 -# NB: Sometimes Anaconda revs the version, in which case you'll have to -# update this! -# An example of this happened on Aug 13, 2019, when osx-64/python-2.7.16-h97142e2_2.tar.bz2 -# was uploaded to https://anaconda.org/anaconda/python/files -if [[ "$desired_python" == 3.5 ]]; then - mac_version='macosx_10_6_x86_64' -elif [[ "$desired_python" == 2.7 ]]; then - mac_version='macosx_10_7_x86_64' -elif [[ -n "$CROSS_COMPILE_ARM64" ]]; then +if [[ -n "$CROSS_COMPILE_ARM64" || $(uname -m) == "arm64" ]]; then mac_version='macosx_11_0_arm64' else mac_version='macosx_10_9_x86_64' @@ -128,7 +119,7 @@ if [[ ! -d "$pytorch_rootdir" ]]; then popd fi pushd "$pytorch_rootdir" -git submodule update --init --recursive --jobs 0 +git submodule update --init --recursive popd ########################## @@ -144,6 +135,11 @@ SETUPTOOLS_PINNED_VERSION="=46.0.0" PYYAML_PINNED_VERSION="=5.3" EXTRA_CONDA_INSTALL_FLAGS="" case ${desired_python} in + 3.11) + SETUPTOOLS_PINNED_VERSION=">=46.0.0" + PYYAML_PINNED_VERSION=">=5.3" + NUMPY_PINNED_VERSION="==1.23.5" + ;; 3.10) SETUPTOOLS_PINNED_VERSION=">=46.0.0" PYYAML_PINNED_VERSION=">=5.3" @@ -167,8 +163,12 @@ tmp_env_name="wheel_py$python_nodot" conda create ${EXTRA_CONDA_INSTALL_FLAGS} -yn "$tmp_env_name" python="$desired_python" source activate "$tmp_env_name" -retry conda install ${EXTRA_CONDA_INSTALL_FLAGS} -yq cmake "numpy${NUMPY_PINNED_VERSION}" nomkl "setuptools${SETUPTOOLS_PINNED_VERSION}" "pyyaml${PYYAML_PINNED_VERSION}" cffi typing_extensions ninja requests -retry conda install ${EXTRA_CONDA_INSTALL_FLAGS} -yq mkl-include==2020.1 mkl-static==2020.1 -c intel +if [[ "$desired_python" == "3.11" ]]; then + retry pip install -q "numpy${NUMPY_PINNED_VERSION}" "setuptools${SETUPTOOLS_PINNED_VERSION}" "pyyaml${PYYAML_PINNED_VERSION}" typing_extensions requests +else + retry conda install ${EXTRA_CONDA_INSTALL_FLAGS} -yq "numpy${NUMPY_PINNED_VERSION}" nomkl "setuptools${SETUPTOOLS_PINNED_VERSION}" "pyyaml${PYYAML_PINNED_VERSION}" typing_extensions requests +fi +retry conda install ${EXTRA_CONDA_INSTALL_FLAGS} -yq cmake ninja mkl-include==2022.2.1 mkl-static==2022.2.1 -c intel retry pip install -qr "${pytorch_rootdir}/requirements.txt" || true # For USE_DISTRIBUTED=1 on macOS, need libuv and pkg-config to find libuv. diff --git a/windows/build_all.bat b/windows/build_all.bat index 0c1edcf655..f60da8c763 100755 --- a/windows/build_all.bat +++ b/windows/build_all.bat @@ -30,8 +30,8 @@ set "ORIG_PATH=%PATH%" conda remove -n py36 --all -y || rmdir %CONDA_HOME%\envs\py36 /s conda remove -n py37 --all -y || rmdir %CONDA_HOME%\envs\py37 /s -conda create -n py36 -y -q numpy=1.11 mkl=2018 cffi pyyaml boto3 cmake ninja typing_extensions python=3.6 -conda create -n py37 -y -q numpy=1.11 mkl=2018 cffi pyyaml boto3 cmake ninja typing_extensions python=3.7 +conda create -n py36 -y -q numpy=1.11 mkl=2018 pyyaml boto3 cmake ninja typing_extensions python=3.6 +conda create -n py37 -y -q numpy=1.11 mkl=2018 pyyaml boto3 cmake ninja typing_extensions python=3.7 REM Install MKL rmdir /s /q mkl diff --git a/windows/condaenv.bat b/windows/condaenv.bat index 470575340f..6d945badd1 100644 --- a/windows/condaenv.bat +++ b/windows/condaenv.bat @@ -9,10 +9,11 @@ FOR %%v IN (%DESIRED_PYTHON%) DO ( set PYTHON_VERSION_STR=%%v set PYTHON_VERSION_STR=!PYTHON_VERSION_STR:.=! conda remove -n py!PYTHON_VERSION_STR! --all -y || rmdir %CONDA_HOME%\envs\py!PYTHON_VERSION_STR! /s - if "%%v" == "3.7" call conda create -n py!PYTHON_VERSION_STR! -y -q numpy=1.11 "mkl=2020.2" cffi pyyaml boto3 cmake ninja typing_extensions python=%%v + if "%%v" == "3.7" call conda create -n py!PYTHON_VERSION_STR! -y -q numpy=1.11 "mkl=2020.2" pyyaml boto3 cmake ninja typing_extensions python=%%v if "%%v" == "3.8" call conda create -n py!PYTHON_VERSION_STR! -y -q numpy=1.11 "mkl=2020.2" pyyaml boto3 cmake ninja typing_extensions python=%%v if "%%v" == "3.9" call conda create -n py!PYTHON_VERSION_STR! -y -q "numpy>=1.11" "mkl=2020.2" pyyaml boto3 cmake ninja typing_extensions python=%%v if "%%v" == "3.10" call conda create -n py!PYTHON_VERSION_STR! -y -q -c=conda-forge "numpy>=1.21.2" "mkl=2020.2" pyyaml boto3 "cmake=3.19.6" ninja typing_extensions python=%%v + if "%%v" == "3.11" call conda create -n py!PYTHON_VERSION_STR! -y -q -c=conda-forge "numpy>=1.21.2" "mkl=2020.2" pyyaml boto3 "cmake=3.19.6" ninja typing_extensions python=%%v if "%%v" == "3" call conda create -n py!PYTHON_VERSION_STR! -y -q numpy=1.11 "mkl=2020.2" pyyaml boto3 cmake ninja typing_extensions python=%%v ) endlocal diff --git a/windows/cuda102.bat b/windows/cuda102.bat deleted file mode 100644 index 1d90c86b81..0000000000 --- a/windows/cuda102.bat +++ /dev/null @@ -1,58 +0,0 @@ -@echo off - -set MODULE_NAME=pytorch - -IF NOT EXIST "setup.py" IF NOT EXIST "%MODULE_NAME%" ( - call internal\clone.bat - cd .. -) ELSE ( - call internal\clean.bat -) -IF ERRORLEVEL 1 goto :eof - -call internal\check_deps.bat -IF ERRORLEVEL 1 goto :eof - -REM Check for optional components - -set USE_CUDA= -set CMAKE_GENERATOR=Visual Studio 15 2017 Win64 - -IF "%NVTOOLSEXT_PATH%"=="" ( - IF EXIST "C:\Program Files\NVIDIA Corporation\NvToolsExt\lib\x64\nvToolsExt64_1.lib" ( - set NVTOOLSEXT_PATH=C:\Program Files\NVIDIA Corporation\NvToolsExt - ) ELSE ( - echo NVTX ^(Visual Studio Extension ^for CUDA^) ^not installed, failing - exit /b 1 - ) -) - -IF "%CUDA_PATH_V10_2%"=="" ( - IF EXIST "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin\nvcc.exe" ( - set "CUDA_PATH_V10_2=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2" - ) ELSE ( - echo CUDA 10.2 not found, failing - exit /b 1 - ) -) - -IF "%BUILD_VISION%" == "" ( - set TORCH_CUDA_ARCH_LIST=3.7+PTX;5.0;6.0;6.1;7.0;7.5 - set TORCH_NVCC_FLAGS=-Xfatbin -compress-all -) ELSE ( - set NVCC_FLAGS=-D__CUDA_NO_HALF_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_50,code=compute_50 -) - -set "CUDA_PATH=%CUDA_PATH_V10_2%" -set "PATH=%CUDA_PATH_V10_2%\bin;%PATH%" - -:optcheck - -call internal\check_opts.bat -IF ERRORLEVEL 1 goto :eof - -call internal\copy.bat -IF ERRORLEVEL 1 goto :eof - -call internal\setup.bat -IF ERRORLEVEL 1 goto :eof diff --git a/windows/cuda115.bat b/windows/cuda115.bat deleted file mode 100644 index bf037b22cc..0000000000 --- a/windows/cuda115.bat +++ /dev/null @@ -1,58 +0,0 @@ -@echo off - -set MODULE_NAME=pytorch - -IF NOT EXIST "setup.py" IF NOT EXIST "%MODULE_NAME%" ( - call internal\clone.bat - cd .. -) ELSE ( - call internal\clean.bat -) -IF ERRORLEVEL 1 goto :eof - -call internal\check_deps.bat -IF ERRORLEVEL 1 goto :eof - -REM Check for optional components - -set USE_CUDA= -set CMAKE_GENERATOR=Visual Studio 15 2017 Win64 - -IF "%NVTOOLSEXT_PATH%"=="" ( - IF EXIST "C:\Program Files\NVIDIA Corporation\NvToolsExt\lib\x64\nvToolsExt64_1.lib" ( - set NVTOOLSEXT_PATH=C:\Program Files\NVIDIA Corporation\NvToolsExt - ) ELSE ( - echo NVTX ^(Visual Studio Extension ^for CUDA^) ^not installed, failing - exit /b 1 - ) -) - -IF "%CUDA_PATH_V115%"=="" ( - IF EXIST "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\bin\nvcc.exe" ( - set "CUDA_PATH_V115=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5" - ) ELSE ( - echo CUDA 11.5 not found, failing - exit /b 1 - ) -) - -IF "%BUILD_VISION%" == "" ( - set TORCH_CUDA_ARCH_LIST=3.7+PTX;5.0;6.0;6.1;7.0;7.5;8.0;8.6 - set TORCH_NVCC_FLAGS=-Xfatbin -compress-all -) ELSE ( - set NVCC_FLAGS=-D__CUDA_NO_HALF_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_86,code=compute_86 -) - -set "CUDA_PATH=%CUDA_PATH_V115%" -set "PATH=%CUDA_PATH_V115%\bin;%PATH%" - -:optcheck - -call internal\check_opts.bat -IF ERRORLEVEL 1 goto :eof - -call internal\copy.bat -IF ERRORLEVEL 1 goto :eof - -call internal\setup.bat -IF ERRORLEVEL 1 goto :eof diff --git a/windows/cuda113.bat b/windows/cuda118.bat similarity index 81% rename from windows/cuda113.bat rename to windows/cuda118.bat index 568f1e754d..02d91adc38 100644 --- a/windows/cuda113.bat +++ b/windows/cuda118.bat @@ -27,24 +27,24 @@ IF "%NVTOOLSEXT_PATH%"=="" ( ) ) -IF "%CUDA_PATH_V113%"=="" ( - IF EXIST "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\bin\nvcc.exe" ( - set "CUDA_PATH_V113=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3" +IF "%CUDA_PATH_V118%"=="" ( + IF EXIST "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin\nvcc.exe" ( + set "CUDA_PATH_V118=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8" ) ELSE ( - echo CUDA 11.3 not found, failing + echo CUDA 11.8 not found, failing exit /b 1 ) ) IF "%BUILD_VISION%" == "" ( - set TORCH_CUDA_ARCH_LIST=3.7+PTX;5.0;6.0;6.1;7.0;7.5;8.0;8.6 + set TORCH_CUDA_ARCH_LIST=3.7+PTX;5.0;6.0;6.1;7.0;7.5;8.0;8.6;9.0 set TORCH_NVCC_FLAGS=-Xfatbin -compress-all ) ELSE ( - set NVCC_FLAGS=-D__CUDA_NO_HALF_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_86,code=compute_86 + set NVCC_FLAGS=-D__CUDA_NO_HALF_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_90,code=compute_90 ) -set "CUDA_PATH=%CUDA_PATH_V113%" -set "PATH=%CUDA_PATH_V113%\bin;%PATH%" +set "CUDA_PATH=%CUDA_PATH_V118%" +set "PATH=%CUDA_PATH_V118%\bin;%PATH%" :optcheck diff --git a/windows/internal/check_deps.bat b/windows/internal/check_deps.bat index 25c4c4a51d..5e1f58e35e 100755 --- a/windows/internal/check_deps.bat +++ b/windows/internal/check_deps.bat @@ -16,18 +16,16 @@ IF "%BUILD_VISION%" == "" ( ) ) -IF NOT EXIST "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" ( - echo Visual Studio 2017 C++ BuildTools is required to compile PyTorch on Windows +if not exist "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" ( + echo Visual Studio %VC_YEAR% C++ BuildTools is required to compile PyTorch on Windows exit /b 1 ) -IF "%VC_YEAR%" == "" set VC_YEAR=2019 - -set VC_VERSION_LOWER=16 -set VC_VERSION_UPPER=17 -IF "%VC_YEAR%" == "2017" ( - set VC_VERSION_LOWER=15 - set VC_VERSION_UPPER=16 +set VC_VERSION_LOWER=17 +set VC_VERSION_UPPER=18 +if "%VC_YEAR%" == "2019" ( + set VC_VERSION_LOWER=16 + set VC_VERSION_UPPER=17 ) if NOT "%VS15INSTALLDIR%" == "" if exist "%VS15INSTALLDIR%\VC\Auxiliary\Build\vcvarsall.bat" ( diff --git a/windows/internal/cuda_install.bat b/windows/internal/cuda_install.bat index a79571014b..b4f11a58a4 100644 --- a/windows/internal/cuda_install.bat +++ b/windows/internal/cuda_install.bat @@ -17,69 +17,23 @@ set CUDNN_FOLDER="cuda" set CUDNN_LIB_FOLDER="lib\x64" :: Skip all of this if we already have cuda installed -if exist "C:\\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v%CUDA_VERSION_STR%\bin\nvcc.exe" goto set_cuda_env_vars +if exist "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v%CUDA_VERSION_STR%\bin\nvcc.exe" goto set_cuda_env_vars -if %CUDA_VER% EQU 102 goto cuda102 -if %CUDA_VER% EQU 113 goto cuda113 -if %CUDA_VER% EQU 115 goto cuda115 if %CUDA_VER% EQU 116 goto cuda116 if %CUDA_VER% EQU 117 goto cuda117 +if %CUDA_VER% EQU 118 goto cuda118 echo CUDA %CUDA_VERSION_STR% is not supported exit /b 1 -:cuda102 - -if not exist "%SRC_DIR%\temp_build\cuda_10.2.89_441.22_win10.exe" ( - curl -k -L https://ossci-windows.s3.amazonaws.com/cuda_10.2.89_441.22_win10.exe --output "%SRC_DIR%\temp_build\cuda_10.2.89_441.22_win10.exe" - if errorlevel 1 exit /b 1 - set "CUDA_SETUP_FILE=%SRC_DIR%\temp_build\cuda_10.2.89_441.22_win10.exe" - set "ARGS=nvcc_10.2 cuobjdump_10.2 nvprune_10.2 cupti_10.2 cublas_10.2 cublas_dev_10.2 cudart_10.2 cufft_10.2 cufft_dev_10.2 curand_10.2 curand_dev_10.2 cusolver_10.2 cusolver_dev_10.2 cusparse_10.2 cusparse_dev_10.2 nvgraph_10.2 nvgraph_dev_10.2 npp_10.2 npp_dev_10.2 nvrtc_10.2 nvrtc_dev_10.2 nvml_dev_10.2" -) - -if not exist "%SRC_DIR%\temp_build\cudnn-10.2-windows10-x64-v7.6.5.32.zip" ( - curl -k -L https://ossci-windows.s3.amazonaws.com/cudnn-10.2-windows10-x64-v7.6.5.32.zip --output "%SRC_DIR%\temp_build\cudnn-10.2-windows10-x64-v7.6.5.32.zip" - if errorlevel 1 exit /b 1 - set "CUDNN_SETUP_FILE=%SRC_DIR%\temp_build\cudnn-10.2-windows10-x64-v7.6.5.32.zip" -) - -goto cuda_common - -:cuda113 - -set CUDA_INSTALL_EXE=cuda_11.3.0_465.89_win10.exe -if not exist "%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" ( - curl -k -L "https://ossci-windows.s3.amazonaws.com/%CUDA_INSTALL_EXE%" --output "%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" - if errorlevel 1 exit /b 1 - set "CUDA_SETUP_FILE=%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" - set "ARGS=thrust_11.3 nvcc_11.3 cuobjdump_11.3 nvprune_11.3 nvprof_11.3 cupti_11.3 cublas_11.3 cublas_dev_11.3 cudart_11.3 cufft_11.3 cufft_dev_11.3 curand_11.3 curand_dev_11.3 cusolver_11.3 cusolver_dev_11.3 cusparse_11.3 cusparse_dev_11.3 npp_11.3 npp_dev_11.3 nvrtc_11.3 nvrtc_dev_11.3 nvml_dev_11.3" -) - -set CUDNN_FOLDER=cudnn-windows-x86_64-8.3.2.44_cuda11.5-archive -set CUDNN_LIB_FOLDER="lib" -set CUDNN_INSTALL_ZIP=%CUDNN_FOLDER%.zip" -if not exist "%SRC_DIR%\temp_build\%CUDNN_INSTALL_ZIP%" ( - curl -k -L "http://s3.amazonaws.com/ossci-windows/%CUDNN_INSTALL_ZIP%" --output "%SRC_DIR%\temp_build\%CUDNN_INSTALL_ZIP%" - if errorlevel 1 exit /b 1 - set "CUDNN_SETUP_FILE=%SRC_DIR%\temp_build\%CUDNN_INSTALL_ZIP%" -) - -@REM Cuda 8.3+ required zlib to be installed on the path -echo Installing ZLIB dlls -curl -k -L "http://s3.amazonaws.com/ossci-windows/zlib123dllx64.zip" --output "%SRC_DIR%\temp_build\zlib123dllx64.zip" -7z x "%SRC_DIR%\temp_build\zlib123dllx64.zip" -o"%SRC_DIR%\temp_build\zlib" -xcopy /Y "%SRC_DIR%\temp_build\zlib\dll_x64\*.dll" "C:\Windows\System32" - -goto cuda_common - -:cuda115 +:cuda116 -set CUDA_INSTALL_EXE=cuda_11.5.0_496.13_win10.exe +set CUDA_INSTALL_EXE=cuda_11.6.0_511.23_windows.exe if not exist "%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" ( curl -k -L "https://ossci-windows.s3.amazonaws.com/%CUDA_INSTALL_EXE%" --output "%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" if errorlevel 1 exit /b 1 set "CUDA_SETUP_FILE=%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" - set "ARGS=thrust_11.5 nvcc_11.5 cuobjdump_11.5 nvprune_11.5 nvprof_11.5 cupti_11.5 cublas_11.5 cublas_dev_11.5 cudart_11.5 cufft_11.5 cufft_dev_11.5 curand_11.5 curand_dev_11.5 cusolver_11.5 cusolver_dev_11.5 cusparse_11.5 cusparse_dev_11.5 npp_11.5 npp_dev_11.5 nvrtc_11.5 nvrtc_dev_11.5 nvml_dev_11.5" + set "ARGS=thrust_11.6 nvcc_11.6 cuobjdump_11.6 nvprune_11.6 nvprof_11.6 cupti_11.6 cublas_11.6 cublas_dev_11.6 cudart_11.6 cufft_11.6 cufft_dev_11.6 curand_11.6 curand_dev_11.6 cusolver_11.6 cusolver_dev_11.6 cusparse_11.6 cusparse_dev_11.6 npp_11.6 npp_dev_11.6 nvrtc_11.6 nvrtc_dev_11.6 nvml_dev_11.6" ) set CUDNN_FOLDER=cudnn-windows-x86_64-8.3.2.44_cuda11.5-archive @@ -99,17 +53,17 @@ xcopy /Y "%SRC_DIR%\temp_build\zlib\dll_x64\*.dll" "C:\Windows\System32" goto cuda_common -:cuda116 +:cuda117 -set CUDA_INSTALL_EXE=cuda_11.6.0_511.23_windows.exe +set CUDA_INSTALL_EXE=cuda_11.7.0_516.01_windows.exe if not exist "%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" ( curl -k -L "https://ossci-windows.s3.amazonaws.com/%CUDA_INSTALL_EXE%" --output "%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" if errorlevel 1 exit /b 1 set "CUDA_SETUP_FILE=%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" - set "ARGS=thrust_11.6 nvcc_11.6 cuobjdump_11.6 nvprune_11.6 nvprof_11.6 cupti_11.6 cublas_11.6 cublas_dev_11.6 cudart_11.6 cufft_11.6 cufft_dev_11.6 curand_11.6 curand_dev_11.6 cusolver_11.6 cusolver_dev_11.6 cusparse_11.6 cusparse_dev_11.6 npp_11.6 npp_dev_11.6 nvrtc_11.6 nvrtc_dev_11.6 nvml_dev_11.6" + set "ARGS=thrust_11.7 nvcc_11.7 cuobjdump_11.7 nvprune_11.7 nvprof_11.7 cupti_11.7 cublas_11.7 cublas_dev_11.7 cudart_11.7 cufft_11.7 cufft_dev_11.7 curand_11.7 curand_dev_11.7 cusolver_11.7 cusolver_dev_11.7 cusparse_11.7 cusparse_dev_11.7 npp_11.7 npp_dev_11.7 nvrtc_11.7 nvrtc_dev_11.7 nvml_dev_11.7" ) -set CUDNN_FOLDER=cudnn-windows-x86_64-8.3.2.44_cuda11.5-archive +set CUDNN_FOLDER=cudnn-windows-x86_64-8.5.0.96_cuda11-archive set CUDNN_LIB_FOLDER="lib" set "CUDNN_INSTALL_ZIP=%CUDNN_FOLDER%.zip" if not exist "%SRC_DIR%\temp_build\%CUDNN_INSTALL_ZIP%" ( @@ -126,17 +80,17 @@ xcopy /Y "%SRC_DIR%\temp_build\zlib\dll_x64\*.dll" "C:\Windows\System32" goto cuda_common -:cuda117 +:cuda118 -set CUDA_INSTALL_EXE=cuda_11.7.0_516.01_windows.exe +set CUDA_INSTALL_EXE=cuda_11.8.0_522.06_windows.exe if not exist "%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" ( curl -k -L "https://ossci-windows.s3.amazonaws.com/%CUDA_INSTALL_EXE%" --output "%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" if errorlevel 1 exit /b 1 set "CUDA_SETUP_FILE=%SRC_DIR%\temp_build\%CUDA_INSTALL_EXE%" - set "ARGS=thrust_11.7 nvcc_11.7 cuobjdump_11.7 nvprune_11.7 nvprof_11.7 cupti_11.7 cublas_11.7 cublas_dev_11.7 cudart_11.7 cufft_11.7 cufft_dev_11.7 curand_11.7 curand_dev_11.7 cusolver_11.7 cusolver_dev_11.7 cusparse_11.7 cusparse_dev_11.7 npp_11.7 npp_dev_11.7 nvrtc_11.7 nvrtc_dev_11.7 nvml_dev_11.7" + set "ARGS=cuda_profiler_api_11.8 thrust_11.8 nvcc_11.8 cuobjdump_11.8 nvprune_11.8 nvprof_11.8 cupti_11.8 cublas_11.8 cublas_dev_11.8 cudart_11.8 cufft_11.8 cufft_dev_11.8 curand_11.8 curand_dev_11.8 cusolver_11.8 cusolver_dev_11.8 cusparse_11.8 cusparse_dev_11.8 npp_11.8 npp_dev_11.8 nvrtc_11.8 nvrtc_dev_11.8 nvml_dev_11.8" ) -set CUDNN_FOLDER=cudnn-windows-x86_64-8.5.0.96_cuda11-archive +set CUDNN_FOLDER=cudnn-windows-x86_64-8.7.0.84_cuda11-archive set CUDNN_LIB_FOLDER="lib" set "CUDNN_INSTALL_ZIP=%CUDNN_FOLDER%.zip" if not exist "%SRC_DIR%\temp_build\%CUDNN_INSTALL_ZIP%" ( @@ -158,7 +112,7 @@ goto cuda_common :: With GHA runners these should be pre-installed as part of our AMI process :: If you cannot find the CUDA version you want to build for here then please :: add it @ https://github.com/pytorch/test-infra/tree/main/aws/ami/windows -if not exist "C:\\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v%CUDA_VERSION_STR%\bin\nvcc.exe" ( +if not exist "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v%CUDA_VERSION_STR%\bin\nvcc.exe" ( if not exist "%SRC_DIR%\temp_build\NvToolsExt.7z" ( curl -k -L https://ossci-windows.s3.us-east-1.amazonaws.com/builder/NvToolsExt.7z --output "%SRC_DIR%\temp_build\NvToolsExt.7z" if errorlevel 1 exit /b 1 @@ -183,12 +137,12 @@ if not exist "C:\\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v%CUDA_VERSION popd echo Installing VS integration... - if "%VC_YEAR%" == "2017" ( - xcopy /Y "%SRC_DIR%\temp_build\cuda\CUDAVisualStudioIntegration\extras\visual_studio_integration\MSBuildExtensions\*.*" "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\Common7\IDE\VC\VCTargets\BuildCustomizations" - ) if "%VC_YEAR%" == "2019" ( xcopy /Y "%SRC_DIR%\temp_build\cuda\CUDAVisualStudioIntegration\extras\visual_studio_integration\MSBuildExtensions\*.*" "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\BuildCustomizations" ) + if "%VC_YEAR%" == "2022" ( + xcopy /Y "%SRC_DIR%\temp_build\cuda\CUDAVisualStudioIntegration\extras\visual_studio_integration\MSBuildExtensions\*.*" "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\BuildCustomizations" + ) echo Installing NvToolsExt... 7z x %SRC_DIR%\temp_build\NvToolsExt.7z -o"%SRC_DIR%\temp_build\NvToolsExt" diff --git a/windows/internal/env_fix.bat b/windows/internal/env_fix.bat index dd0aaf5f2d..2a53198a99 100644 --- a/windows/internal/env_fix.bat +++ b/windows/internal/env_fix.bat @@ -5,12 +5,19 @@ setlocal -IF NOT EXIST "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" ( - echo Visual Studio 2017 C++ BuildTools is required to compile PyTorch on Windows +if not exist "%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" ( + echo Visual Studio %VC_YEAR% C++ BuildTools is required to compile PyTorch on Windows exit /b 1 ) -for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -products * -version [15^,16^) -property installationPath`) do ( +set VC_VERSION_LOWER=17 +set VC_VERSION_UPPER=18 +if "%VC_YEAR%" == "2019" ( + set VC_VERSION_LOWER=16 + set VC_VERSION_UPPER=17 +) + +for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -products * -version [%VC_VERSION_LOWER%^,%VC_VERSION_UPPER%^) -property installationPath`) do ( if exist "%%i" if exist "%%i\VC\Auxiliary\Build\vcvarsall.bat" ( set "VS15INSTALLDIR=%%i" set "VS15VCVARSALL=%%i\VC\Auxiliary\Build\vcvarsall.bat" @@ -20,8 +27,8 @@ for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio :vswhere -IF "%VS15VCVARSALL%"=="" ( - echo Visual Studio 2017 C++ BuildTools is required to compile PyTorch on Windows +if "%VS15VCVARSALL%"=="" ( + echo Visual Studio %VC_YEAR% C++ BuildTools is required to compile PyTorch on Windows exit /b 1 ) diff --git a/windows/internal/install_nightly_package.bat b/windows/internal/install_nightly_package.bat deleted file mode 100644 index 7db23ebd60..0000000000 --- a/windows/internal/install_nightly_package.bat +++ /dev/null @@ -1,67 +0,0 @@ -if "%PACKAGE_TYPE%" == "wheel" goto wheel -if "%PACKAGE_TYPE%" == "conda" goto conda -if "%PACKAGE_TYPE%" == "libtorch" goto libtorch - -:wheel -echo "install pytorch wheel from nightly" - -set pip_url="https://download.pytorch.org/whl/nightly/%DESIRED_CUDA%/torch_nightly.html" -if "%DESIRED_CUDA%" == "cu102" ( - set package_name_and_version="torch==%NIGHTLIES_DATE_PREAMBLE%%DATE%" -) else ( - set package_name_and_version="torch==%NIGHTLIES_DATE_PREAMBLE%%DATE%+%DESIRED_CUDA%" -) -pip install "%package_name_and_version%" -f "%pip_url%" --no-cache-dir --no-index -q -if errorlevel 1 exit /b 1 - -exit /b 0 - -:conda -echo "install pytorch conda from nightly" -set package_name_and_version="pytorch==%NIGHTLIES_DATE_PREAMBLE%%DATE%" - -if "%DESIRED_CUDA%" == "cpu" ( - call conda install -yq -c pytorch-nightly %package_name_and_version% cpuonly -) else ( - call conda install -yq -c pytorch-nightly "cudatoolkit=%CUDA_VERSION_STR%" %package_name_and_version% -) -if ERRORLEVEL 1 exit /b 1 - -FOR /f %%i in ('python -c "import sys;print(sys.version)"') do set cur_python=%%i - -if not %cur_python:~0,3% == %DESIRED_PYTHON% ( - echo "The Python version has changed to %cur_python%" - echo "Probably the package for the version we want does not exist" - echo "conda will change the Python version even if it was explicitly declared" -) - -if "%DESIRED_CUDA%" == "cpu" ( - call conda list torch | findstr cuda || exit /b 0 - echo "The installed package is built for CUDA, the full package is" - call conda list torch -) else ( - call conda list torch | findstr cuda%CUDA_VERSION% && exit /b 0 - echo "The installed package doesn't seem to be built for CUDA "%CUDA_VERSION_STR% - echo "the full package is " - call conda list torch -) -exit /b 1 - -:libtorch -echo "install libtorch from nightly" -if "%LIBTORCH_CONFIG%" == "debug" ( - set NAME_PREFIX=libtorch-win-shared-with-deps-debug -) else ( - set NAME_PREFIX=libtorch-win-shared-with-deps -) -if "%DESIRED_CUDA%" == "cu102" ( - set package_name=%NAME_PREFIX%-%NIGHTLIES_DATE_PREAMBLE%%DATE%.zip -) else ( - set package_name=%NAME_PREFIX%-%NIGHTLIES_DATE_PREAMBLE%%DATE%%%2B%DESIRED_CUDA%.zip -) -set libtorch_url="https://download.pytorch.org/libtorch/nightly/%DESIRED_CUDA%/%package_name%" -curl --retry 3 -k "%libtorch_url%" -o %package_name% -if ERRORLEVEL 1 exit /b 1 - -7z x %package_name% -otmp -if ERRORLEVEL 1 exit /b 1 diff --git a/windows/internal/smoke_test.bat b/windows/internal/smoke_test.bat index 836a04311b..2e1b1b243a 100644 --- a/windows/internal/smoke_test.bat +++ b/windows/internal/smoke_test.bat @@ -30,6 +30,7 @@ exit /b 1 echo "install wheel package" set PYTHON_INSTALLER_URL= +if "%DESIRED_PYTHON%" == "3.11" set "PYTHON_INSTALLER_URL=https://www.python.org/ftp/python/3.11.0/python-3.11.0-amd64.exe" if "%DESIRED_PYTHON%" == "3.10" set "PYTHON_INSTALLER_URL=https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe" if "%DESIRED_PYTHON%" == "3.9" set "PYTHON_INSTALLER_URL=https://www.python.org/ftp/python/3.9.0/python-3.9.0-amd64.exe" if "%DESIRED_PYTHON%" == "3.8" set "PYTHON_INSTALLER_URL=https://www.python.org/ftp/python/3.8.2/python-3.8.2-amd64.exe" @@ -51,13 +52,8 @@ set "PATH=%CD%\Python%PYTHON_VERSION%\Scripts;%CD%\Python;%PATH%" pip install -q numpy protobuf "mkl>=2019" if errorlevel 1 exit /b 1 -if "%TEST_NIGHTLY_PACKAGE%" == "1" ( - call internal\install_nightly_package.bat - if errorlevel 1 exit /b 1 -) else ( - for /F "delims=" %%i in ('where /R "%PYTORCH_FINAL_PACKAGE_DIR:/=\%" *.whl') do pip install "%%i" - if errorlevel 1 exit /b 1 -) +for /F "delims=" %%i in ('where /R "%PYTORCH_FINAL_PACKAGE_DIR:/=\%" *.whl') do pip install "%%i" +if errorlevel 1 exit /b 1 goto smoke_test @@ -68,15 +64,15 @@ echo "install conda package" set "CONDA_HOME=%CD%\conda" set "tmp_conda=%CONDA_HOME%" set "miniconda_exe=%CD%\miniconda.exe" -set "CONDA_EXTRA_ARGS=" -if "%CUDA_VERSION%" == "115" ( - set "CONDA_EXTRA_ARGS=-c=nvidia" -) +set "CONDA_EXTRA_ARGS=cpuonly -c pytorch-nightly" if "%CUDA_VERSION%" == "116" ( - set "CONDA_EXTRA_ARGS=-c=nvidia" + set "CONDA_EXTRA_ARGS=pytorch-cuda=11.6 -c nvidia -c pytorch-nightly" ) if "%CUDA_VERSION%" == "117" ( - set "CONDA_EXTRA_ARGS=-c=nvidia" + set "CONDA_EXTRA_ARGS=pytorch-cuda=11.7 -c nvidia -c pytorch-nightly" +) +if "%CUDA_VERSION%" == "118" ( + set "CONDA_EXTRA_ARGS=pytorch-cuda=11.8 -c nvidia -c pytorch-nightly" ) rmdir /s /q conda @@ -93,9 +89,8 @@ if errorlevel 1 exit /b 1 call %CONDA_HOME%\condabin\activate.bat testenv if errorlevel 1 exit /b 1 -call conda update -n base -y -c defaults conda - -call conda install %CONDA_EXTRA_ARGS% -yq protobuf numpy +:: do conda install to make sure all the dependencies are installed +call conda install -yq pytorch %CONDA_EXTRA_ARGS% if ERRORLEVEL 1 exit /b 1 set /a CUDA_VER=%CUDA_VERSION% @@ -103,25 +98,8 @@ set CUDA_VER_MAJOR=%CUDA_VERSION:~0,-1% set CUDA_VER_MINOR=%CUDA_VERSION:~-1,1% set CUDA_VERSION_STR=%CUDA_VER_MAJOR%.%CUDA_VER_MINOR% -if "%TEST_NIGHTLY_PACKAGE%" == "1" ( - call internal\install_nightly_package.bat - if errorlevel 1 exit /b 1 - goto smoke_test -) - -for /F "delims=" %%i in ('where /R "%PYTORCH_FINAL_PACKAGE_DIR:/=\%" *.tar.bz2') do call conda install %CONDA_EXTRA_ARGS% -y "%%i" --offline -if ERRORLEVEL 1 exit /b 1 - -if "%CUDA_VERSION%" == "cpu" goto install_cpu_torch - -:: We do an update --all here since that will install the dependencies for any package that's installed offline -call conda update --all %CONDA_EXTRA_ARGS% -y -c pytorch -c defaults -c numba/label/dev -if ERRORLEVEL 1 exit /b 1 - -goto smoke_test - -:install_cpu_torch -call conda install %CONDA_EXTRA_ARGS% -y cpuonly -c pytorch +:: Install package we just build +for /F "delims=" %%i in ('where /R "%PYTORCH_FINAL_PACKAGE_DIR:/=\%" *.tar.bz2') do call conda install -yq "%%i" --offline if ERRORLEVEL 1 exit /b 1 :smoke_test @@ -162,24 +140,21 @@ goto end :libtorch echo "install and test libtorch" -if "%VC_YEAR%" == "2017" powershell internal\vs2017_install.ps1 +if "%VC_YEAR%" == "2019" powershell internal\vs2019_install.ps1 +if "%VC_YEAR%" == "2022" powershell internal\vs2022_install.ps1 + if ERRORLEVEL 1 exit /b 1 -if "%TEST_NIGHTLY_PACKAGE%" == "1" ( - call internal\install_nightly_package.bat - if errorlevel 1 exit /b 1 -) else ( - for /F "delims=" %%i in ('where /R "%PYTORCH_FINAL_PACKAGE_DIR:/=\%" *-latest.zip') do 7z x "%%i" -otmp - if ERRORLEVEL 1 exit /b 1 -) +for /F "delims=" %%i in ('where /R "%PYTORCH_FINAL_PACKAGE_DIR:/=\%" *-latest.zip') do 7z x "%%i" -otmp +if ERRORLEVEL 1 exit /b 1 pushd tmp\libtorch -set VC_VERSION_LOWER=16 -set VC_VERSION_UPPER=17 -IF "%VC_YEAR%" == "2017" ( - set VC_VERSION_LOWER=15 - set VC_VERSION_UPPER=16 +set VC_VERSION_LOWER=17 +set VC_VERSION_UPPER=18 +IF "%VC_YEAR%" == "2019" ( + set VC_VERSION_LOWER=16 + set VC_VERSION_UPPER=17 ) for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -legacy -products * -version [%VC_VERSION_LOWER%^,%VC_VERSION_UPPER%^) -property installationPath`) do ( @@ -192,7 +167,7 @@ for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio :vswhere IF "%VS15VCVARSALL%"=="" ( - echo Visual Studio 2017 C++ BuildTools is required to compile PyTorch test on Windows + echo Visual Studio %VC_YEAR% C++ BuildTools is required to compile PyTorch test on Windows exit /b 1 ) call "%VS15VCVARSALL%" x64 @@ -202,13 +177,13 @@ set INCLUDE=%INCLUDE%;%install_root%\include;%install_root%\include\torch\csrc\a set LIB=%LIB%;%install_root%\lib set PATH=%PATH%;%install_root%\lib -cl %BUILDER_ROOT%\test_example_code\simple-torch-test.cpp c10.lib torch_cpu.lib /EHsc +cl %BUILDER_ROOT%\test_example_code\simple-torch-test.cpp c10.lib torch_cpu.lib /EHsc /std:c++17 if ERRORLEVEL 1 exit /b 1 .\simple-torch-test.exe if ERRORLEVEL 1 exit /b 1 -cl %BUILDER_ROOT%\test_example_code\check-torch-mkl.cpp c10.lib torch_cpu.lib /EHsc +cl %BUILDER_ROOT%\test_example_code\check-torch-mkl.cpp c10.lib torch_cpu.lib /EHsc /std:c++17 if ERRORLEVEL 1 exit /b 1 .\check-torch-mkl.exe @@ -223,9 +198,9 @@ set BUILD_SPLIT_CUDA= if exist "%install_root%\lib\torch_cuda_cu.lib" if exist "%install_root%\lib\torch_cuda_cpp.lib" set BUILD_SPLIT_CUDA=ON if "%BUILD_SPLIT_CUDA%" == "ON" ( - cl %BUILDER_ROOT%\test_example_code\check-torch-cuda.cpp torch_cpu.lib c10.lib torch_cuda_cu.lib torch_cuda_cpp.lib /EHsc /link /INCLUDE:?warp_size@cuda@at@@YAHXZ /INCLUDE:?_torch_cuda_cu_linker_symbol_op_cuda@native@at@@YA?AVTensor@2@AEBV32@@Z + cl %BUILDER_ROOT%\test_example_code\check-torch-cuda.cpp torch_cpu.lib c10.lib torch_cuda_cu.lib torch_cuda_cpp.lib /EHsc /std:c++17 /link /INCLUDE:?warp_size@cuda@at@@YAHXZ /INCLUDE:?_torch_cuda_cu_linker_symbol_op_cuda@native@at@@YA?AVTensor@2@AEBV32@@Z ) else ( - cl %BUILDER_ROOT%\test_example_code\check-torch-cuda.cpp torch_cpu.lib c10.lib torch_cuda.lib /EHsc /link /INCLUDE:?warp_size@cuda@at@@YAHXZ + cl %BUILDER_ROOT%\test_example_code\check-torch-cuda.cpp torch_cpu.lib c10.lib torch_cuda.lib /EHsc /std:c++17 /link /INCLUDE:?warp_size@cuda@at@@YAHXZ ) .\check-torch-cuda.exe if ERRORLEVEL 1 exit /b 1 diff --git a/windows/internal/vc_install_helper.bat b/windows/internal/vc_install_helper.bat index 6a2a0e0d99..61ab6d5f8c 100644 --- a/windows/internal/vc_install_helper.bat +++ b/windows/internal/vc_install_helper.bat @@ -1,7 +1,12 @@ if "%VC_YEAR%" == "2019" powershell windows/internal/vs2019_install.ps1 +if "%VC_YEAR%" == "2022" powershell windows/internal/vs2022_install.ps1 -set VC_VERSION_LOWER=16 -set VC_VERSION_UPPER=17 +set VC_VERSION_LOWER=17 +set VC_VERSION_UPPER=18 +if "%VC_YEAR%" == "2019" ( + set VC_VERSION_LOWER=16 + set VC_VERSION_UPPER=17 +) for /f "usebackq tokens=*" %%i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -products Microsoft.VisualStudio.Product.BuildTools -version [%VC_VERSION_LOWER%^,%VC_VERSION_UPPER%^) -property installationPath`) do ( if exist "%%i" if exist "%%i\VC\Auxiliary\Build\vcvarsall.bat" ( diff --git a/windows/internal/vs2017_install.ps1 b/windows/internal/vs2017_install.ps1 deleted file mode 100644 index 873e4eb17f..0000000000 --- a/windows/internal/vs2017_install.ps1 +++ /dev/null @@ -1,28 +0,0 @@ -$VS_DOWNLOAD_LINK = "https://aka.ms/vs/15/release/vs_buildtools.exe" -$VS_INSTALL_ARGS = @("--nocache","--quiet","--wait", "--add Microsoft.VisualStudio.Workload.VCTools", - "--add Microsoft.Component.MSBuild", - "--add Microsoft.VisualStudio.Component.Roslyn.Compiler", - "--add Microsoft.VisualStudio.Component.TextTemplating", - "--add Microsoft.VisualStudio.Component.VC.CoreIde", - "--add Microsoft.VisualStudio.Component.VC.Redist.14.Latest", - "--add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core", - "--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64", - "--add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Win81") - -if ($args.Count -ne 0) { - $VS_INSTALL_ARGS += "--add Microsoft.VisualStudio.Component.VC.Tools.$($args[0])" -} - -curl.exe --retry 3 -kL $VS_DOWNLOAD_LINK --output vs_installer.exe -if ($LASTEXITCODE -ne 0) { - echo "Download of the VS 2017 installer failed" - exit 1 -} - -$process = Start-Process "${PWD}\vs_installer.exe" -ArgumentList $VS_INSTALL_ARGS -NoNewWindow -Wait -PassThru -Remove-Item -Path vs_installer.exe -Force -$exitCode = $process.ExitCode -if (($exitCode -ne 0) -and ($exitCode -ne 3010)) { - echo "VS 2017 installer exited with code $exitCode, which should be one of [0, 3010]." - exit 1 -} diff --git a/windows/internal/vs2022_install.ps1 b/windows/internal/vs2022_install.ps1 new file mode 100644 index 0000000000..55fba47378 --- /dev/null +++ b/windows/internal/vs2022_install.ps1 @@ -0,0 +1,56 @@ +# https://developercommunity.visualstudio.com/t/install-specific-version-of-vs-component/1142479 +# https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-history#evergreen-bootstrappers + +# 17.4.3 BuildTools +$VS_DOWNLOAD_LINK = "https://download.visualstudio.microsoft.com/download/pr/8f480125-28b8-4a2c-847c-c2b02a8cdd1b/64be21d4ada005d7d07896ed0b004c322409bd04d6e8eba4c03c9fa39c928e7a/vs_BuildTools.exe" +$COLLECT_DOWNLOAD_LINK = "https://aka.ms/vscollect.exe" +$VS_INSTALL_ARGS = @("--nocache","--quiet","--wait", "--add Microsoft.VisualStudio.Workload.VCTools", + "--add Microsoft.Component.MSBuild", + "--add Microsoft.VisualStudio.Component.Roslyn.Compiler", + "--add Microsoft.VisualStudio.Component.TextTemplating", + "--add Microsoft.VisualStudio.Component.VC.CoreIde", + "--add Microsoft.VisualStudio.Component.VC.Redist.14.Latest", + "--add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core", + "--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64", + "--add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Win81") + +curl.exe --retry 3 -kL $VS_DOWNLOAD_LINK --output vs_installer.exe +if ($LASTEXITCODE -ne 0) { + echo "Download of the VS $VC_YEAR Version $VS_VERSION installer failed" + exit 1 +} + +if (Test-Path "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe") { + $existingPath = & "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe" -products "Microsoft.VisualStudio.Product.BuildTools" -version "[17, 18)" -property installationPath + if ($existingPath -ne $null) { + if (!${env:CIRCLECI}) { + echo "Found correctly versioned existing BuildTools installation in $existingPath" + exit 0 + } + echo "Found existing BuildTools installation in $existingPath" + $VS_UNINSTALL_ARGS = @("uninstall", "--installPath", "`"$existingPath`"", "--quiet","--wait") + $process = Start-Process "${PWD}\vs_installer.exe" -ArgumentList $VS_UNINSTALL_ARGS -NoNewWindow -Wait -PassThru + $exitCode = $process.ExitCode + if (($exitCode -ne 0) -and ($exitCode -ne 3010)) { + echo "Original BuildTools uninstall failed with code $exitCode" + exit 1 + } + echo "Original BuildTools uninstalled" + } +} + +$process = Start-Process "${PWD}\vs_installer.exe" -ArgumentList $VS_INSTALL_ARGS -NoNewWindow -Wait -PassThru +Remove-Item -Path vs_installer.exe -Force +$exitCode = $process.ExitCode +if (($exitCode -ne 0) -and ($exitCode -ne 3010)) { + echo "VS $VC_YEAR installer exited with code $exitCode, which should be one of [0, 3010]." + curl.exe --retry 3 -kL $COLLECT_DOWNLOAD_LINK --output Collect.exe + if ($LASTEXITCODE -ne 0) { + echo "Download of the VS Collect tool failed." + exit 1 + } + Start-Process "${PWD}\Collect.exe" -NoNewWindow -Wait -PassThru + New-Item -Path "C:\w\build-results" -ItemType "directory" -Force + Copy-Item -Path "C:\Users\circleci\AppData\Local\Temp\vslogs.zip" -Destination "C:\w\build-results\" + exit 1 +} diff --git a/windows/internal/vs_install.bat b/windows/internal/vs_install.bat index 624227f0be..221ec33136 100644 --- a/windows/internal/vs_install.bat +++ b/windows/internal/vs_install.bat @@ -1,12 +1,12 @@ @echo off -set VS_DOWNLOAD_LINK=https://aka.ms/vs/15/release/vs_buildtools.exe +set VS_DOWNLOAD_LINK=https://download.visualstudio.microsoft.com/download/pr/8f480125-28b8-4a2c-847c-c2b02a8cdd1b/64be21d4ada005d7d07896ed0b004c322409bd04d6e8eba4c03c9fa39c928e7a/vs_BuildTools.exe IF "%VS_LATEST%" == "1" ( set VS_INSTALL_ARGS= --nocache --norestart --quiet --wait --add Microsoft.VisualStudio.Workload.VCTools set VSDEVCMD_ARGS= ) ELSE ( set VS_INSTALL_ARGS=--nocache --quiet --wait --add Microsoft.VisualStudio.Workload.VCTools ^ - --add Microsoft.VisualStudio.Component.VC.Tools.14.11 ^ + --add Microsoft.VisualStudio.Component.VC.Tools.14.34 ^ --add Microsoft.Component.MSBuild ^ --add Microsoft.VisualStudio.Component.Roslyn.Compiler ^ --add Microsoft.VisualStudio.Component.TextTemplating ^ @@ -14,9 +14,9 @@ IF "%VS_LATEST%" == "1" ( --add Microsoft.VisualStudio.Component.VC.Redist.14.Latest ^ --add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core ^ --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 ^ - --add Microsoft.VisualStudio.Component.VC.Tools.14.11 ^ + --add Microsoft.VisualStudio.Component.VC.Tools.14.34 ^ --add Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Win81 - set VSDEVCMD_ARGS=-vcvars_ver=14.11 + set VSDEVCMD_ARGS=-vcvars_ver=14.34 ) curl -k -L %VS_DOWNLOAD_LINK% --output vs_installer.exe