-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bug 1622800 - part 5: Schedule a single screenshot job to Bitrise #6337
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
--- | ||
|
||
loader: taskgraph.loader.transform:loader | ||
|
||
transforms: | ||
- taskgraph.transforms.docker_image:transforms | ||
- taskgraph.transforms.cached_tasks:transforms | ||
- taskgraph.transforms.task:transforms | ||
|
||
jobs: | ||
screenshots: {} | ||
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
--- | ||
loader: taskgraph.loader.transform:loader | ||
|
||
transforms: | ||
- ffios_taskgraph.transforms.screenshots:transforms | ||
- ffios_taskgraph.transforms.secrets:transforms | ||
- taskgraph.transforms.job:transforms | ||
- taskgraph.transforms.task:transforms | ||
|
||
|
||
job-defaults: | ||
description: Generate localized screenshots by delegating the work to bitrise.io | ||
run: | ||
using: run-commands | ||
use-caches: false | ||
secrets: | ||
by-level: | ||
'3': | ||
- name: project/mobile/firefox-ios/bitrise | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I created this secret and put my personal token in it. This is the process I used https://devcenter.bitrise.io/getting-started/account-security/#generating-personal-access-tokens-manually |
||
key: api_key | ||
path: .bitrise_token | ||
default: [] | ||
dummy-secrets: | ||
by-level: | ||
'3': [] | ||
default: | ||
- content: "faketoken" | ||
path: .bitrise_token | ||
run-on-tasks-for: [] | ||
worker: | ||
artifacts: | ||
- type: file | ||
name: public/logs/bitrise.log | ||
path: /builds/worker/checkouts/src/bitrise.log | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This task downloads the actual bitrise logs and re-expose them in Taskcluster. Shell colors are even supported, check this out! https://firefox-ci-tc.services.mozilla.com/tasks/dFVHYucpRjOJEcdQp_OVNQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FdFVHYucpRjOJEcdQp_OVNQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Fbitrise.log |
||
docker-image: {in-tree: screenshots} | ||
max-run-time: 3600 | ||
worker-type: b-linux | ||
|
||
jobs: | ||
fr: | ||
locale: fr | ||
worker: | ||
# TODO: Move artifacts in transforms once we start supporting all locales | ||
artifacts: | ||
- type: file | ||
name: public/screenshots/fr.zip | ||
path: /builds/worker/checkouts/src/fr.zip |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This file is a copy of https://github.com/mozilla-mobile/fenix/blob/24596fd3cd493f38e1212e6ab2844e723565af9b/taskcluster/docker/base/Dockerfile. I changed the python part to be python 3 and to install the required dependencies. Nothing else has changed. |
||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
|
||
FROM ubuntu:18.04 | ||
|
||
MAINTAINER Johan Lorenzo "[email protected]" | ||
|
||
# Add worker user | ||
RUN mkdir /builds && \ | ||
useradd -d /builds/worker -s /bin/bash -m worker && \ | ||
chown worker:worker /builds/worker && \ | ||
mkdir /builds/worker/artifacts && \ | ||
chown worker:worker /builds/worker/artifacts | ||
|
||
WORKDIR /builds/worker/ | ||
|
||
|
||
ENV CURL='curl --location --retry 5' \ | ||
LANG='en_US.UTF-8' \ | ||
TERM='dumb' | ||
|
||
|
||
RUN apt-get update -qq \ | ||
# We need to install tzdata before all of the other packages. Otherwise it will show an interactive dialog that | ||
# we cannot navigate while building the Docker image. | ||
&& apt-get install -y tzdata \ | ||
&& apt-get install -y curl \ | ||
git \ | ||
locales \ | ||
mercurial \ | ||
python3 \ | ||
python3-pip \ | ||
&& apt-get clean | ||
|
||
RUN pip3 install --upgrade pip | ||
COPY requirements.txt ./ | ||
RUN pip3 install -r requirements.txt | ||
|
||
RUN locale-gen "$LANG" | ||
|
||
# %include-run-task | ||
|
||
ENV SHELL=/bin/bash \ | ||
HOME=/builds/worker \ | ||
PATH="/builds/worker/.local/bin:$PATH" | ||
|
||
|
||
VOLUME /builds/worker/checkouts | ||
VOLUME /builds/worker/.cache | ||
|
||
|
||
# run-task expects to run as root | ||
USER root |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
aiohttp-retry | ||
taskcluster |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -57,6 +57,7 @@ def configure_run_commands_schema(config, job, taskdesc): | |
|
||
def _generate_secret_command(secret): | ||
secret_command = [ | ||
"python3", # XXX Other mobile projects run this script under python2 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As far as I know, there is nothing to worry about today. I just left a reminder in case we hit a weird python3 issue someday. |
||
"taskcluster/scripts/get-secret.py", | ||
"-s", secret["name"], | ||
"-k", secret["key"], | ||
|
@@ -70,6 +71,7 @@ def _generate_secret_command(secret): | |
|
||
def _generate_dummy_secret_command(secret): | ||
secret_command = [ | ||
"python3", # XXX Other mobile projects run this script under python2 | ||
"taskcluster/scripts/write-dummy-secret.py", | ||
"-f", secret["path"], | ||
"-c", secret["content"], | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
""" | ||
Resolve secrets and dummy secrets | ||
""" | ||
|
||
from __future__ import absolute_import, print_function, unicode_literals | ||
|
||
from taskgraph.transforms.base import TransformSequence | ||
|
||
|
||
transforms = TransformSequence() | ||
|
||
|
||
@transforms.add | ||
def add_command(config, tasks): | ||
for task in tasks: | ||
commands = task["run"].setdefault("commands", []) | ||
locale = task.pop("locale") | ||
commands.append([ | ||
"python3", | ||
"taskcluster/scripts/generate-screenshots.py", | ||
"--token-file", ".bitrise_token", | ||
"--branch", config.params["head_ref"], | ||
"--commit", config.params["head_rev"], | ||
"--locale", locale | ||
]) | ||
|
||
yield task |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
""" | ||
Resolve secrets and dummy secrets | ||
""" | ||
|
||
from __future__ import absolute_import, print_function, unicode_literals | ||
|
||
from taskgraph.transforms.base import TransformSequence | ||
from taskgraph.util.schema import resolve_keyed_by | ||
|
||
|
||
transforms = TransformSequence() | ||
|
||
|
||
@transforms.add | ||
def resolve_keys(config, tasks): | ||
for task in tasks: | ||
for key in ("run.secrets", "run.dummy-secrets"): | ||
resolve_keyed_by( | ||
task, | ||
key, | ||
item_name=task["name"], | ||
level=config.params["level"] | ||
) | ||
yield task |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This empty dict just means we're building the screenshots docker image with the default values. That is to say: it'll find
taskcluster/docker/screenshots/Dockerfile
and will know what to do with it.