romp

Run on multiple platforms

I use pip-tools but also want to keep my compiled requirements for multiple platforms. Instead of sharing directories between machines or otherwise shuttling files around I chose to create romp which will let me submit arbitrary work to Azure Pipelines to get access to multiple platforms. Personally, I will use this primarily behind boots.

Below is an example usage. It will run a single job under Linux with 64-bit CPython 3.6. The job will execute echo red > blue.txt and will collect *.txt from the job and save it locally to artifacts.tar.gz.

$ venv/bin/romp --platform linux --interpreter cpython --version 3.6 --architecture x86_64 --command 'echo red > blue.txt' --artifact-paths '*.txt' --artifact artifacts.tar.gz
Requesting build
Waiting for build: https://dev.azure.com/altendky/f1722f91-62fe-4a15-8937-252c96b31292/_build/results?buildId=2938
Handling artifact

Since romp leverages Azure Pipelines to get access to all the platforms, server-side setup is required. You will need an Azure account and to create a pipeline (build) within that account using a romp repository (official or your own fork). Configure the pipeline to use deployed_ci/azure/azure-pipelines-lock.yml. It is configured such that commits do not trigger builds. romp's own CI and testing is driven by azure-pipelines.yml in the project root but this file can be ignored for regular use. Once the pipeline is setup you will need to create a Personal Access Token (PAT) to use for authentication when running romp. The PAT will need the build read and execute scope enabled.

For local setup, the command line options can be set by environment variables. For many options this will not make sense but for a few it will. Specifically consider the follow options which will often be consistent for all calls.

• ROMP_BUILD_REQUEST_URL

  • 'https://dev.azure.com/altendky/romp-on/_apis/build/builds?api-version=5.0'

• ROMP_DEFINITION_ID

  • 5

• ROMP_PERSONAL_ACCESS_TOKEN

  • fp6al3jxta2zliz6rh5hr6ewd5nw2hsmasse2laiyoyg7otneqjq

• ROMP_USERNAME

  • altendky

$ venv/bin/romp --help
Usage: romp [OPTIONS]

Options:
  --personal-access-token, --pat TEXT
                                  A personal access token (PAT) with rights to
                                  initiate builds
                                  ($ROMP_PERSONAL_ACCESS_TOKEN)
  --build-request-url TEXT        The URL for submitting a build request
                                  ($ROMP_BUILD_REQUEST_URL)  [default: https:/
                                  /dev.azure.com/altendky/romp/_apis/build/bui
                                  lds?api-version=5.0]
  --command TEXT                  The command to be run for each target
                                  ($ROMP_COMMAND)  [default: python -c 'import
                                  sys; print(sys.version);
                                  print(sys.platform)']
  --username TEXT                 Username for build URL authentication
                                  ($ROMP_USERNAME)  [default: altendky]
  --environments TEXT             Targets to run on.  Mostly use the matrix
                                  options instead.  This may be removed.
                                  ($ROMP_ENVIRONMENTS)
  --check-period INTEGER          The period used to poll the build for
                                  completion ($ROMP_CHECK_PERIOD)  [default:
                                  15]
  --source-branch TEXT            The romp source branch to use for the build
                                  ($ROMP_SOURCE_BRANCH)  [default: develop]
  --definition-id INTEGER         The definition id of the build to be
                                  triggered ($ROMP_DEFINITION_ID)  [default:
                                  3]
  --archive-file FILENAME         The archive to be uploaded to the build
                                  ($ROMP_ARCHIVE)
  --artifact FILENAME             The path at which to save the resulting
                                  artifact ($ROMP_ARTIFACT_PATH)
  --artifact-paths TEXT           Paths on remote system to build the artifact
                                  archive from.  Wildcards are supported via
                                  bash. ($ROMP_ARTIFACT_PATHS)
  --platform [Linux|macOS|Windows]
                                  Platforms to matrix across
                                  ($ROMP_MATRIX_PLATFORMS)  [default: Linux,
                                  macOS, Windows]
  --interpreter [CPython|PyPy]    Interpreters to matrix across
                                  ($ROMP_MATRIX_INTERPRETERS)  [default:
                                  CPython, PyPy]
  --version [2.7|3.4|3.5|3.6|3.7]
                                  Versions to matrix across
                                  ($ROMP_MATRIX_VERSIONS)  [default: 2.7, 3.4,
                                  3.5, 3.6, 3.7]
  --architecture [x86|x86_64]     Architectures to matrix across
                                  ($ROMP_MATRIX_ARCHITECTURES)  [default: x86,
                                  x86_64]
  --include <PLATFORM INTERPRETER VERSION ARCHITECTURE>
                                  Complete environments to include in the
                                  matrix ($ROMP_MATRIX_INCLUDES)
  --exclude <PLATFORM INTERPRETER VERSION ARCHITECTURE>
                                  Complete environments to exclude from the
                                  matrix ($ROMP_MATRIX_EXCLUDES)
  --archive-paths-root DIRECTORY  Files in the uploaded archive will be stored
                                  with paths relative to this path.
                                  ($ROMP_ARCHIVE_PATHS_ROOT)
  --archive-path TEXT             Files to include in the archive which will
                                  be extracted prior on the remote system
                                  prior to running the remote command.
                                  ($ROMP_ARCHIVE_PATHS)
  --verbose                       Increase logging verbosity by up to 2 levels
                                  ($ROMP_VERBOSITY)
  --help                          Show this message and exit.